Skip to content

Cleanup split between lexer and sliding-text-window #79205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 96 commits into from
Jul 23, 2025
Merged

Cleanup split between lexer and sliding-text-window #79205

merged 96 commits into from
Jul 23, 2025

Conversation

CyrusNajmabadi
Copy link
Member

The general philosophy here is that the SlidingTextWindow is just a representation of characters, and doesn't really care anything about "current lexeme". Instead, that entire concept it contained entirely within the lexer itself.

This helps allow us to tweak these components with less risk of regression (like the recent regression around lexing out a dot and ensuring it was not a .. in a range).

CyrusNajmabadi
CyrusNajmabadi commented Jun 30, 2025
View reviewed changes
src/Compilers/CSharp/Portable/Parser/AbstractLexer.cs
@@ -15,6 +13,7 @@ internal class AbstractLexer : IDisposable
{
internal readonly SlidingTextWindow TextWindow;
private List<SyntaxDiagnosticInfo>? _errors;
protected int LexemeStartPosition;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved to lexer-layer from SlidingTextWindow

CyrusNajmabadi
CyrusNajmabadi commented Jun 30, 2025
View reviewed changes
src/Compilers/CSharp/Portable/Parser/AbstractLexer.cs
=> TextWindow.GetText(LexemeStartPosition, intern: false);

protected string GetInternedLexemeText()
=> TextWindow.GetText(LexemeStartPosition, intern: true);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these .GetText helpers on the TextWindow were used all over the place. MOved to two simple helpers on Lexer for getting either an interned or non-interned chunk of text for the current lexeme.

CyrusNajmabadi
CyrusNajmabadi commented Jun 30, 2025
View reviewed changes
src/Compilers/CSharp/Portable/Parser/AbstractLexer.cs
=> TextWindow.GetText(LexemeStartPosition, intern: true);

protected int CurrentLexemeWidth
=> this.TextWindow.Position - LexemeStartPosition;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was previously exposed as the (imo) confusingly named SlidingTextWindow.Width.

I tried to ensure that helpers related to lexemes used that term in their name to keep it clear what 'width/count/etc.' things are referring to.

CyrusNajmabadi
CyrusNajmabadi commented Jun 30, 2025
View reviewed changes
src/Compilers/CSharp/Portable/Parser/LexerCache.cs
internal SyntaxTrivia LookupWhitespaceTrivia(
SlidingTextWindow textWindow,
int lexemeStartPosition,
int hashCode)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

easier to just read as the new function. There was only ever one caller ofr LookupTrivia, and it always used the same generic arg.

CyrusNajmabadi
CyrusNajmabadi commented Jun 30, 2025
View reviewed changes
src/Compilers/CSharp/Portable/Parser/Lexer_RawStringLiteral.cs Outdated
@@ -87,7 +87,7 @@ private void ScanRawStringLiteral(ref TokenInfo info, bool inDirective)
// trusting the contents.
if (this.HasErrors)
{
var afterStartDelimiter = TextWindow.LexemeStartPosition + startingQuoteCount;
var afterStartDelimiter = this.LexemeStartPosition + startingQuoteCount;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reference to textWindow.LexemeStartPosition were mechanically replaced with this.LexemeStartPosition

CyrusNajmabadi
CyrusNajmabadi commented Jun 30, 2025
View reviewed changes
src/Compilers/CSharp/Portable/Parser/Lexer_RawStringLiteral.cs Outdated
@@ -120,7 +120,7 @@ private void ScanRawStringLiteral(ref TokenInfo info, bool inDirective)
};
}

info.Text = TextWindow.GetText(intern: true);
info.Text = this.GetInternedLexemeText();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

References to TextWindow.GetText(intern: true) and TextWindow.GetText(intern: false) were mechanically updated to this.GetInternedLexemeText() and this.GetNonInternedLexemeText() mechanically.

CyrusNajmabadi
CyrusNajmabadi commented Jun 30, 2025
View reviewed changes
src/Compilers/CSharp/Portable/Parser/QuickScanner.cs Outdated

// The max index in charWindow that we will quick scan to. This is either the end of the window
// or the position of the largest token we'd be willing to quick scan and cache.
var maxIndexInWindow = Math.Min(charWindow.Length, startIndexInWindow + MaxCachedTokenSize);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed these all for clarity.

@CyrusNajmabadi
Copy link
Member Author

@dotnet/roslyn-compiler for another pair of eyes.

@CyrusNajmabadi
Copy link
Member Author

@dotnet/roslyn-compiler for another pair of eyes. thanks!

jjonescz
jjonescz reviewed Jul 23, 2025
View reviewed changes
src/Compilers/CSharp/Portable/Parser/SlidingTextWindow.cs Outdated
_textEnd = text.Length;
_strings = StringTable.GetInstance();
_characterWindow = s_windowPool.Allocate();
_lexemeStart = 0;
_characterWindow = new(s_windowPool.Allocate());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think target-typed new() is disallowed in the compiler layer.

src/Compilers/CSharp/Portable/Parser/SlidingTextWindow.cs Outdated
return _characterWindow;
}
}
public readonly ReadOnlySpan<char> CurrentWindowSpan => _characterWindow.AsSpan(_positionInText - _characterWindowStartPositionInText);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that _positionInText and the character window might not always intersect. For example, Advance methods only move the position but not the window. Could it happen that the AsSpan would fail then? E.g., because the offset would be negative (position would be before the window).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Good call. I will fix that up.

src/Compilers/CSharp/Portable/Parser/QuickScanner.cs
var charPropLength = CharProperties.Length;

for (; i < n; i++)
// Where we are currently pointing in the charWindow as we read in a character at a time.
var currentIndex = 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we put this inside the for initializer?

Copy link
Member Author

@CyrusNajmabadi CyrusNajmabadi Jul 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no. it is read below (it is used to figure out how many characters were actually read out).

CyrusNajmabadi
CyrusNajmabadi commented Jul 23, 2025
View reviewed changes
CyrusNajmabadi
CyrusNajmabadi commented Jul 23, 2025
View reviewed changes
Copy link
Member Author

@CyrusNajmabadi CyrusNajmabadi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

c

src/Compilers/CSharp/Portable/Parser/SlidingTextWindow.cs Outdated
return _characterWindow;
}
}
public readonly ReadOnlySpan<char> CurrentWindowSpan => _characterWindow.AsSpan(_positionInText - _characterWindowStartPositionInText);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Good call. I will fix that up.

CyrusNajmabadi
CyrusNajmabadi commented Jul 23, 2025
View reviewed changes
Copy link
Member Author

@CyrusNajmabadi CyrusNajmabadi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

c

CyrusNajmabadi

This comment was marked as off-topic.

Sign in to view

CyrusNajmabadi

This comment was marked as off-topic.

Sign in to view

@CyrusNajmabadi CyrusNajmabadi requested a review from jjonescz July 23, 2025 09:10
jjonescz
jjonescz reviewed Jul 23, 2025
View reviewed changes
src/Compilers/CSharp/Portable/Parser/SlidingTextWindow.cs Outdated
{
get
{
return _positionInText > _textEnd
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't look like the right check. If position is after end, that's shouldn't cause problems. But the _positionInText - _characterWindowStartPositionInText offset computed below can be negative if _characterWindowStartPositionInText is after _positionInText, so I would expect this check to be something like:

Suggested change
return _positionInText > _textEnd
return _positionInText < _characterWindowStartPositionInText

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both woudl actually cause problems (since AsSpan checks start/end pos). I'll validate both sides :)

@CyrusNajmabadi CyrusNajmabadi enabled auto-merge (squash) July 23, 2025 10:16
@CyrusNajmabadi CyrusNajmabadi merged commit a39c638 into dotnet:main Jul 23, 2025
23 of 24 checks passed
@CyrusNajmabadi CyrusNajmabadi deleted the lexerField branch July 23, 2025 11:18
@dotnet-policy-service dotnet-policy-service bot added this to the Next milestone Jul 23, 2025
333fred added a commit that referenced this pull request Jul 24, 2025
@333fred @CyrusNajmabadi
@AlekseyTs @jcouv @JoeRobich @jjonescz @dotnet-maestro @akhera99 @tmat @ToddGrun @dibarbet @jasonmalinowski @davidwengier @DoctorKrolic @mmitche @dkurepa @Copilot
Merge main to runtime async branch (#79582)
caeac86 
* Support interpolated string handlers in extension blocks (#78425)

* Support interpolated string handlers in extension blocks

Closes #78137.

* A few tests fail PEVerify

* Update handling of receivers in static scenarios and add tests for indexers/static methods

* Update after merge

* Ensure nullable index adjustment happens always, not just when the receiver parameter is used.

* Additional il verification progress

* PR feedback. The only remaining item is additional testing for ref safety to record the current state.

* Additional PR feedback

* More testing, suggested simplification

* Simplify and more testing.

* Feedback

* Suggested optimization

* Add skipped test for #78433.

* Fix loss of nullability attributes when getting the 'constructed reduced from' method. (#79400)

* Add tracking issue for disabled assert (#79427)

Reference tracking issue #79426 for the disabled assert.

* Fix code gen for some compound assignment scenarios involving extension properties. (#79339)

* Extensions: extension marker type name (#79308)

* Move to xunit.runner.visualstudio 3.1.3

* Avoid eliding pointer-to-ref conversions (#79311)

* Avoid eliding pointer to ref conversions

* Consider only user-defined locals

* Update pre-existing tests

* Test evaluation order

* Explain user-defined local check

* Avoid reusing temps whose refs might be captured (#76009)

* Avoid reusing temps whose refs might be captured

* Revert some changes

* Simplify the heuristic

* Avoid reusing any local whose address has been taken

* Update tests

* Inline a function

* Revert unrelated change

* Remove non-reusable locals after checking for them

* Keep ref count for addressed locals list

* Extend an assert

* Add high-level comment to MightEscapeTemporaryRefs

* Filter non-reusable locals

* Simplify `int`s to `bool`s

* Replace coalesce with an assert

* Mark nested calls as always used

* Fix this parameter of nint methods

* Test chained call

* Simplify by using this parameter symbol

* Fix readonly targets

* Fixup a test

* Remove an unused parameter

* Fix nullability after merge

* [main] Update dependencies from dotnet/arcade (#79385)

[main] Update dependencies from dotnet/arcade


 - Revert .NET SDK bump

 - Set NUGET_PACKAGES for bootstrap builds in CI

 - Set NUGET_PACKAGES for Correctness_Build_Artifacts

 - Add tracking issue for workarounds

 - Call nuget restore with the -publish flag

 - We need to restore where we publish

 - Add comment for strange restore behavior

 - Update comment

* Fixes rename attribute bug (#79418)

* fix bug

* update test

* simplify

* comment

* clarity

* gracefully handle the document or languageservice being null

* Adds EqualityComparer<T>.Create polyfill

* Revert compiler changes

* Update dependencies from https://github.com/dotnet/arcade build 20250717.3

Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.XliffTasks
 From Version 10.0.0-beta.25358.3 -> To Version 10.0.0-beta.25367.3

* Move to .NET 10p6 (#79438)

* Move to .NET 10p6

Let's try to get ahead of the next one.

* Update contributing docs as well.

* Allow razor to use SolutionChecksumUpdater (#79373)

* Allow razor to use SolutionChecksumUpdater

SolutionChecksumUpdater allows incremental updates of sourcetexts in the server process, but was previously limited to just documents in the solution. Instead, by loosening this to allow TextDocuments, razor editing can have incremental updates of the primary razor document when cohosting is turned on.

* use is/or instead of ==/||

* Remove dependency on ClassificationOptions and DefinitionItem from Semantic Search service (#79439)

* Update insertions for VS release changes

* Fix static extension method not showing up on enum type

* Add workitem

* cleanup

* Use ConfigureAwait(true) when in a blocking JTF run call

* GetTextSynchronously

* Update src/VisualStudio/Core/Def/LanguageService/AbstractLanguageService`2.VsLanguageDebugInfo.cs

Co-authored-by: Jason Malinowski <[email protected]>

* Make synchronous

* Docs

* Allow Razor to create a formatting options, and send it to OOP

* Update dependencies from https://github.com/dotnet/arcade build 20250717.4

Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.XliffTasks
 From Version 10.0.0-beta.25358.3 -> To Version 10.0.0-beta.25367.4

* Remove assumption about a tree shape that compound assignment produces (#79432)

* Skip GetProjectXml_FileBasedProgram_SdkTooOld_01 test

* Bump Microsoft.VisualStudio.Extensibility.Testing.Xunit to 0.1.796-beta

This brings along some fixes needed for it to work under newer versions
of xunit.runner.visualstudio.

* Reduce path length of Roslyn ServiceHub Services folder (#79460)

I've been hitting lots of gold bars in VS indicating various Roslyn features are unavailable when running when F5'ing the roslyn project. I finally spent some time digging into why this happens on my machine, and it turns out that out service hub service paths are just a bit too long. This PR just reduces that path length by 14 characters, which is enough for all the services to be under the 260 character long path limit.

* downgrade SDK

* Fix Typo

* Simplify stale project handling (#79386)

* Update System.Memory to prevent bootstrap failure

* Include dependencies of System.Memory in VSIX deployment project

* remove unnecessary additional reference

* Update dependencies from https://github.com/dotnet/arcade build 20250717.5

Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.XliffTasks
 From Version 10.0.0-beta.25358.3 -> To Version 10.0.0-beta.25367.5

* Use new compiler API

* Correctly change return type of partial definition part

* Feedback

* Move RoslynParallel and ProducerConsumer to threading source package (#79474)

* Downgrade SDK

* Do not remove usings/imports in code-cleanup when file has syntax errors

* Add testws

* Find add-using results in SG docs

* Docs

* Do not place SuppressMessage attributes on an extension block

* remove asynchrony from service registration

* Ignore all targets of MS.CA.Threading.Package in rebuild (#79495)

* Don't mark special compiler synthesized members as obsolete

* Add test

* Use globs in .projitems for source packages (#79480)

* Fix name declaration completion in primary constructor parameters

* nrt

* nrt

* Generate complogs on build failure (#79446)

* Generate complogs on build failure

This should make it easier to reproduce CI compiler failures locally.

* Test bad change for complog creation

* Use --yes

* Include full exception

* Revert "Test bad change for complog creation"

This reverts commit a465981.

* remove from pkgdefs

* Extensions: fix issue with GetParameterOrExtensionParameter (#79402)

* Fix exception when requesting additional file diagnostics

* Do not do full cleanup when we are formatting a new document

The CodeAction.CleanupDocumentAsync was added in
#59091 to ensure the document
had elastic trivia formatted before we applied the result to the
file. It appears the intent was to do just the trivia formatting, but
the CleanupDocumentAsync does a lot of expensive operations around
simplification and adding imports, which aren't needed in this case.

Fixes https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1540532

* Extensions: add test for function type scenario (#79323)

* Reenable analyzer that was disabled when we moved to .Net 10 Preview 5

* Allow captured primary constructor parameters to be still used in base-list

* Simplify ancestor checks

* Fix 'this' keyword not being recomended where allowed

* in progress

* Fix 'switch' keyword recomendations after an expression

* Fix escaping of enum members in completion

* Suggest proper 'Async'-suffixed name for Task-returning symbols

* Fix location of 'partial' modifier when generating new partial members

* Update VMR orchestrator property (#79517)

* Make attachdebugger flag work on non-Windows (#79499)

These compiler flags used Debugger.Launch, which only works on windows. I borrowed the equivalent code from the LSP launcher, which already has this functionality (https://github.com/dotnet/roslyn/blob/cf34acbc5d0f64ff87013cdd02ae61cff562daaf/src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/Program.cs#L75-L89).

* Downgrade System.Memory to make VS happy

* Fix auth issue in PR val builds

* Fix

* Update Basic.Reference.Assemblies (#79525)

* Update Basic.Reference.Assemblies

Update to version 1.8.3 of Basic.Reference.Assemblies and make references to .NET 10p6 available in our tests.

* Feedback

* Feedback

* Allow add using to work on Razor generated documents

* Improve compiler server/client logging (#79335)

* Remove unused constant

* Include timestamp in logs

* Log messages to msbuild as well

* Log server's process ID

* Log process creation on Unix

* Exclude some messages from the binlog

* Fix test loggers

* Revert "Exclude some messages from the binlog"

This reverts commit 12c1050.

* Cleanup

* Simplify

* Simplify

* Simplify

* Add tests

* Add tests

* Simplify

* Simplify

* initial change

* Renames

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Extensions: address or split remaining open issues directly associated with test plan (#79452)

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* Update tests

* In progress

* in progress

* in progress

* in progress

* in progress

* in progress

* in progress

* in progress

* Cleanup split between lexer and sliding-text-window (#79205)

Co-authored-by: Jan Jones <[email protected]>

* Add docs

* Fix name qualification when moving a using alias outside of a file scoped namespace

* Fix strucutre out of bounds

* Tests

* Add Version.Details.props (#79539)

* Fix issue with 'use explicit type' and nullable tuples

* Add back

* Convert ValueTaskFactory methods to static extensions (#79541)

* Update logic

* Also downgrade System.Numerics.Vectors for VS

* Add Enum static extensions (#79546)

* Fix main build break

* Fix main build break AGAIN

* Projects to deploy (#79430)

* [main] Source code updates from dotnet/dotnet (#79483)

[main] Source code updates from dotnet/dotnet


 - Workaround cref resolve issue due to new RuntimeHelpers in SCI

* [main] Source code updates from dotnet/dotnet (#79563)

[main] Source code updates from dotnet/dotnet

* Revert "Fix main build break AGAIN (#79559)"

This reverts commit e67e2a4, reversing
changes made to 1a74e66.

* Tweak build break fix

* Tweak build break fix

* inline test strings

* Cleanup test methods

* Fix issue where we weren't properly adding elastic trivia to newly generated members

* Fix behavior

* Add test

* Mark SyntaxTokenParser as non-experimental (#79521)

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

---------

Co-authored-by: Cyrus Najmabadi <[email protected]>
Co-authored-by: AlekseyTs <[email protected]>
Co-authored-by: Julien Couvreur <[email protected]>
Co-authored-by: Joey Robichaud <[email protected]>
Co-authored-by: Jan Jones <[email protected]>
Co-authored-by: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Joey Robichaud <[email protected]>
Co-authored-by: Ankita Khera <[email protected]>
Co-authored-by: tmat <[email protected]>
Co-authored-by: Tomáš Matoušek <[email protected]>
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Todd Grunke <[email protected]>
Co-authored-by: David Barbet <[email protected]>
Co-authored-by: Cyrus Najmabadi <[email protected]>
Co-authored-by: Jason Malinowski <[email protected]>
Co-authored-by: David Wengier <[email protected]>
Co-authored-by: Jason Malinowski <[email protected]>
Co-authored-by: DoctorKrolic <[email protected]>
Co-authored-by: Matt Mitchell <[email protected]>
Co-authored-by: Jan Jones <[email protected]>
Co-authored-by: Djuradj Kurepa <[email protected]>
Co-authored-by: Copilot <[email protected]>
This was referenced Jul 29, 2025
Closed
Closed
@RikkiGibson RikkiGibson modified the milestones: Next, 18.0 P1 Aug 19, 2025
@dotnet-bot dotnet-bot mentioned this pull request Aug 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@333fred 333fred 333fred approved these changes

@jjonescz jjonescz jjonescz approved these changes

Assignees
No one assigned
Labels
Area-Compilers
Projects
None yet
Milestone
18.0 P1
Development

Successfully merging this pull request may close these issues.
4 participants
@CyrusNajmabadi @333fred @jjonescz @RikkiGibson