Incorrect Colorization for Custom Iterator Type

[ArktisDev]

Bug type: Language Service

Describe the bug

• OS and Version: MacOS Big Sur 11.4
• VS Code Version: 1.66.0
• C/C++ Extension Version: 1.9.7
• Other extensions you installed (and if the issue persists after disabling them): None

Workspace is a single C++ file containing a definition of a custom std::forward_iterator_tag implementation. Colorization of this is incorrect when used in a for loop as in below picture:

Steps to reproduce
Open MWE with setting "C_Cpp.enhancedColorization": "Enabled".

Expected behavior
Brackets are colorized as normal.

Code sample and logs

• Code sample

#include <iostream>

class Foo {
    public:
        int a;
        
        Foo(int a) : a( a ) {}
};

class FooIterator : public std::forward_iterator_tag {
    public:
        int a;
        const Foo* f;
        
        FooIterator(int a, const Foo& f) : a( a ), f( &f ) {}
        
        FooIterator operator++() {
            advance();
            return *this;
        }
        
        bool operator==(const FooIterator& f) const {
            return this->a == f.a;
        }
        
        bool operator!=(const FooIterator& f) const {
            return this->a != f.a;
        }

        int operator*() const {
            return a;
        }
        
        void advance() {
            a++;
        }
};

FooIterator begin(const Foo& f) {
    return FooIterator(0, f);
}

FooIterator end(const Foo& f) {
    return FooIterator(f.a, f);
}

int main() {
    Foo f(10);
    
    for (int a : f) {
        std::cout << a << std::endl;
    }
    
    return 0;
}

• Configurations in c_cpp_properties.json

{
    "configurations": [
        {
            "name": "Mac",
            "includePath": [
                "${workspaceFolder}/**"
            ],
            "defines": [],
            "macFrameworkPath": [
                "/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/System/Library/Frameworks"
            ],
            "compilerPath": "/usr/bin/g++",
            "cStandard": "c17",
            "cppStandard": "c++20"
        }
    ],
    "version": 4
}

• Logs from running C/C++: Log Diagnostics from the VS Code command palette

-------- Diagnostics - 3/31/2022, 3:25:00 PM
Version: 1.9.7
Current Configuration:
{
    "name": "Mac",
    "includePath": [
        "${workspaceFolder}/**"
    ],
    "defines": [],
    "macFrameworkPath": [
        "/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/System/Library/Frameworks"
    ],
    "compilerPath": "/usr/bin/g++",
    "cStandard": "c17",
    "cppStandard": "c++20",
    "compilerPathIsExplicit": true,
    "cStandardIsExplicit": true,
    "cppStandardIsExplicit": true,
    "intelliSenseModeIsExplicit": false,
    "compilerArgs": [],
    "mergeConfigurations": false,
    "browse": {
        "path": [
            "${workspaceFolder}/**"
        ],
        "limitSymbolsToIncludedHeaders": true
    }
}
Translation Unit Mappings:
[ /Users/Lars/Desktop/ColoringIssue/main.cpp ]:
    /Users/Lars/Desktop/ColoringIssue/main.cpp
Translation Unit Configurations:
[ /Users/Lars/Desktop/ColoringIssue/main.cpp ]:
    Process ID: 34495
    Memory Usage: 139 MB
    Compiler Path: /usr/bin/g++
    Includes:
        /usr/local/include
        /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include/c++/v1
        /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5/include
        /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include
        /Library/Developer/CommandLineTools/usr/include
    Frameworks:
        /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/System/Library/Frameworks
    Standard Version: c++20
    IntelliSense Mode: macos-clang-x64
    Other Flags:
        --clang
        --clang_version=110000
Total Memory Usage: 139 MB

------- Workspace parsing diagnostics -------
Number of files discovered (not excluded): 30097

• Logs from the language server logging

loggingLevel: Debug
loggingLevel has changed to: Debug
cpptools/didChangeCppProperties
$/setTraceNotification
cpptools/getCodeActions: /Users/Lars/Desktop/ColoringIssue/main.cpp (id: 82)
cpptools/textEditorSelectionChange
cpptools/textEditorSelectionChange
cpptools/activeDocumentChange: /Users/Lars/Desktop/ColoringIssue/main.cpp
cpptools/getFoldingRanges: /Users/Lars/Desktop/ColoringIssue/main.cpp (id: 83)
cpptools/textEditorSelectionChange
cpptools/textEditorSelectionChange
cpptools/activeDocumentChange: /Users/Lars/Desktop/ColoringIssue/main.cpp
willSaveWaitUntil: 0ms
textDocument/didSave: /Users/Lars/Desktop/ColoringIssue/main.cpp
  tag parsing file: /Users/Lars/Desktop/ColoringIssue/main.cpp
cpptools/fileChanged: /Users/Lars/Desktop/ColoringIssue/main.cpp
idle loop: reparsing the active document
Checking for syntax errors: /Users/Lars/Desktop/ColoringIssue/main.cpp
Queueing IntelliSense update for files in translation unit of: /Users/Lars/Desktop/ColoringIssue/main.cpp
cpptools/finishUpdateSquiggles
Error squiggle count: 0
Update IntelliSense time (sec): 0.258
cpptools/getSemanticTokens: /Users/Lars/Desktop/ColoringIssue/main.cpp (id: 85)
textDocument/didClose: /Users/Lars/Desktop/ColoringIssue/main.cpp
cpptools/getCodeActions: /Users/Lars/Desktop/ColoringIssue/main.cpp (id: 86)
textDocument/didOpen: /Users/Lars/Desktop/ColoringIssue/main.cpp
Checking for syntax errors: /Users/Lars/Desktop/ColoringIssue/main.cpp
Queueing IntelliSense update for files in translation unit of: /Users/Lars/Desktop/ColoringIssue/main.cpp
cpptools/textEditorSelectionChange
cpptools/getDocumentSymbols: /Users/Lars/Desktop/ColoringIssue/main.cpp (id: 87)
cpptools/textEditorSelectionChange
cpptools/getSemanticTokens: /Users/Lars/Desktop/ColoringIssue/main.cpp (id: 88)
cpptools/activeDocumentChange: /Users/Lars/Desktop/ColoringIssue/main.cpp
cpptools/getDocumentSymbols
cpptools/getFoldingRanges: /Users/Lars/Desktop/ColoringIssue/main.cpp (id: 89)
textDocument/hover: /Users/Lars/Desktop/ColoringIssue/main.cpp (id: 90)
Request canceled: 90
cpptools/finishUpdateSquiggles
Error squiggle count: 0
terminating child process: 34560
Update IntelliSense time (sec): 1.05
cpptools/getFoldingRanges: /Users/Lars/Desktop/ColoringIssue/main.cpp (id: 91)

Screenshots

Looks like the tokens given to the bracket are very wrong:

as compared to a normal for loop:

[sean-mcmanus]

You're getting the semantic token that is after the f. If you put your cursor on the f you'll see its semantic is the correct type of "variable". Not sure why it is giving "function" after the f, but it shouldn't matter since there's no actual token at that spot.

[sean-mcmanus]

Oops, I see now -- the ) { is colored yellow. Yeah, it's a bug.

[sean-mcmanus]

I've filed a bug on our VS shared parser: 1512200