Persistently cached declaration maps

[gabritto]

Follow up to #1767.

In that PR, we computed documentPositionMappers based on declaration map files and cached them at the language service level. This can be rather inefficient because a language service is created per request, so the cached position mappers were recomputed for every LS request.

In this PR, we cache position mappers in the source file structures themselves, and those get persisted across snapshots and therefore across different language services. We achieve this in the following way:

1. When we get a request in the server, we get an LS from session that is based on the current snapshot. We use this LS to compute the response.
2. We collect all file paths present in the response which could be subjected to declaration mapping.
3. We ask session for a new LS that is based on the old snapshot but contains declaration maps for the files collected in the previous step.
4. We compute a new response by mapping old positions to new positions using this new LS with declaration map information.

To implement step 3, we now have a new operation on snapshots, CloneWithSourceMaps, which clones a snapshot with additional declaration map information. This is done by creating a snapshotFSBuilder based on the current snapshot, and having the snapshot FS builder read additional files and compute declaration map information.

To persist these newly read files and declaration map information across future snapshots, we also add these files and computed information to the session's current snapshot. This is implemented in CloneWithDiskChanges. We also update file watchers to include any newly read disk file that was added as part of those changes.

Note: a snapshot FS has two kinds of files cached, overlays and disk files. Overlays are files open in the client, and their contents may or may not match the corresponding disk files, if those exist. To simplify the implementation here, we only compute declaration map information based on disk files. That means if a .d.ts file or .d.ts.map file is open in the client, we won't use the overlay contents, and will instead read from disk. If the overlay content matches the disk content, there's no noticeable difference. If it doesn't, there will be a mismatch in the mapped positions. But in this scenario, something has gone wrong already: if the user edits a .d.ts file in the client (but doesn't save it to disk), and that .d.ts had a declaration map generated for it, the mapping of positions is already going to be wrong.

[gabritto]

This is needed to avoid crashes in the tests, since a file name for a bundled file doesn't round trip with the current implementations of FileNameToDocumentURI and documentUri.FileName().

[gabritto]

I'm planning on making this generic when we add position mapping for other requests.

[gabritto]

The argument for why it's safe to call file.ECMALineInfo() here is kind of subtle, but basically boils down to the fact that, during the process of computing a new snapshot with added map info, we only ever read files or add the computed map info to existing files. We never change a file's content.

[gabritto]

This is not atomic or following any concurrency safe pattern because we don't need to: these are always computed sequentially for now.

[Copilot]

Pull Request Overview

This PR implements persistently cached declaration maps to improve performance. Previously, declaration maps were computed at the language service level and recomputed for each LS request, making the system inefficient.

Key changes include:

• Caching source map information in source file structures that persist across snapshots
• Adding CloneWithSourceMaps and CloneWithDiskChanges operations to snapshots
• Modifying the definition handler to use source maps for accurate position mapping

Reviewed Changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
internal/sourcemap/source_mapper.go	Split source map parsing logic to separate URL detection from mapper creation
internal/project/snapshotfs.go	Added source map computation and caching logic to snapshot file system
internal/project/snapshot.go	Added new snapshot cloning operations for source maps and disk changes
internal/project/session.go	Updated language service creation to return snapshots and handle mapped files
internal/ls/definition.go	Modified definition handling to extract files for mapping and apply source maps
internal/ls/source_map.go	Updated source mapping logic to work with new architecture
internal/ls/languageservice.go	Simplified to delegate source map operations to host
internal/ls/host.go	Updated interface to provide document position mappers
internal/lsp/server.go	Added specialized definition handler with source map support
internal/project/watch.go	Extended file watching patterns to include declaration maps
testdata/baselines/reference/fourslash/goToDefinition/declarationMapGoToDefinitionChanges.baseline.jsonc	Added test baseline for declaration map functionality

[andrewbranch]

Perf nit: AppendIfUnique is an order of magnitude slower than building a map[string]struct{} and converting to a slice by the time N=100.

[andrewbranch]

More importantly, I was expecting to see a tspath.IsDeclarationFileName condition somewhere around here. Is there ever a case where we would need to try to map a location except when it’s a declaration file?

[andrewbranch]

I don’t think it saves a ton of work, but couldn't you do

s.snapshotMu.Lock()
if snapshot == s.snapshot {
  s.snapshot = snapshotWithFiles
} else {
  s.backgroundQueue.Enqueue(...)
}
s.snapshotMu.Unlock()

? It seems like the typical case will be that the session snapshot doesn’t change during source map acquisition, so you can just adopt the new one instead of applying the diff in another clone.

[andrewbranch]

How/when/why does a source mapper point to multiple files? I'm having trouble wrapping my head around what this means.

[andrewbranch]

There can only be changes to source map files if they were deleted by the cleanup logic above, right? The cleanup logic is gated to only run on file open just so most snapshot clones never need to iterate every disk file, so I don’t think we want/need this here. (I bet if you look at snapshot cloning time on keystrokes in large projects, it gets noticeably slower with this.) Do we really need to diff the disk files at all here? It seems like you can just add something more direct in the cleanup logic where the file gets deleted.

[andrewbranch]

I think a similar thing applies here; s.fs.diskFiles can be very large, and I think we usually expect changes to be relatively small, right? It would probably be much more efficient to compute extraDiskFiles from changes rather than diffing the full file cache.

[jakebailey]

Do we know in general how much faster this is compared to the more naive but simple approach? I was kind of hoping we could get away with "caching within a single request" rather than retaining this memory in later snapshots...

[gabritto]

Do we know in general how much faster this is compared to the more naive but simple approach? I was kind of hoping we could get away with "caching within a single request" rather than retaining this memory in later snapshots...

I'm working on a comparison, just happened to find a bug while doing this, so it's taking a while.