feat(lsp): use bulk workspace/diagnostic pull instead of per-file file-walk in runWorkspaceDiagnostics

[apmantza]

Summary

runWorkspaceDiagnostics (clients/lsp/index.ts) computes project-wide diagnostics by walking every LSP-supported source file under the root (collectWorkspaceDiagnosticFiles) and issuing a per-file touchFile(... {diagnostics:"document", clientScope:"all"}) across WORKSPACE_DIAGNOSTICS_CONCURRENCY workers. Its own comment calls this "intentionally expensive." It is — every file must be opened (didOpen) and individually pulled/awaited.

The LSP spec (3.17+) defines a bulk workspace/diagnostic pull: a single request that returns diagnostics for the entire project from the server's already-built model, without opening every file. tsserver, pyright/basedpyright, and rust-analyzer all advertise diagnosticProvider.workspaceDiagnostics: true.

We already detect the capability — client.ts populates workspaceDiagnosticsSupport.mode = "pull" and applyDynamicCapabilities honors a dynamic workspace/diagnostic registration — but we never issue the bulk request. The detected capability is only used to choose per-file pull-vs-push; the project-wide path still brute-forces the file walk.

Proposal

Add a workspaceDiagnostic() client method that issues a real workspace/diagnostic request (with previousResultIds for the incremental report form), and have runWorkspaceDiagnostics prefer it when the active server advertises workspaceDiagnostics, falling back to the current file-walk when it does not (or for files no server claims).

Expected win: project-wide diagnostics without N didOpens — the server returns the whole report it has already computed. Also sidesteps the read-your-writes freshness gymnastics (relates to #179; #180 closed) since the server owns the project model.

Notes / caveats

• Servers may return RelatedFullDocumentDiagnosticReport vs unchanged partials — need result-id caching to exploit the incremental form.
• Some servers stream via $/progress (partial result) — the client must assemble partials.
• Coverage gap: bulk pull only covers files the server's project model includes; mixed-language roots still need the per-file fallback for the other languages. Keep the walk as the fallback, not a removal.
• Source: LSP 3.18 feature-gap audit. Cheap sibling wins (typeDefinition, declaration) already shipped; this is the high-value, higher-effort item left.

Acceptance

• workspaceDiagnostic() client method (full + incremental report forms, partial-result assembly)
• runWorkspaceDiagnostics prefers bulk pull when advertised, falls back to the file walk otherwise
• result-id cache for the unchanged/incremental path
• tests: bulk-pull preferred when advertised; fallback when not; mixed-language root still covered

[apmantza]

Use-case notes (beyond the obvious tool path)

The direct consumer is lens_diagnostics --full (pilens_diagnostics project mode, tools/lens-diagnostics.ts → runWorkspaceDiagnostics) — i.e. this swaps the expensive per-file file-walk for a bulk pull there. That mode already merges LSP results with a runner-based project snapshot + delta, so #235 makes the LSP half cheap/complete, not a new tool.

The higher-leverage uses are where cheapness changes what is feasible to run automatically:

1. Upgrade the error-debt baseline (strongest, half-built). errorDebtBaseline (clients/runtime-coordinator.ts) is today two coarse booleans — tests pass? / build pass? at session start — used at turn-end to nag about added debt (runtime-turn.ts). It is crude precisely because a rich baseline is expensive. A cheap bulk pull lets the baseline become a real per-file diagnostic set captured at session_start; turn-end/session-end then diffs against it and surfaces only net-new errors the agent introduced — including breakage in files it never opened (the cross-file signal it can''t see today).

2. Ground-truth cascade — measured complement to feat(project-intelligence): LSP-confirmed edges in the review graph (provenance-stamped, lazy on cascade hot path) #236. feat(project-intelligence): LSP-confirmed edges in the review graph (provenance-stamped, lazy on cascade hot path) #236 predicts blast radius from the graph; a workspace pull measures the actual post-edit project error set. Pairing = predict on the hot path, periodically reconcile against a cheap bulk pull so the prediction can''t silently drift.

3. Feed the existing ProjectDiagnostic delta (feat(project-diagnostics): normalize remaining turn-end/project runners into ProjectDiagnostic adapters #179). The ProjectDiagnostic snapshot + delta system (writeProjectDiagnosticsDeltaReport / loadProjectDiagnosticsDeltaReport) is runner-powered today; feat(lsp): use bulk workspace/diagnostic pull instead of per-file file-walk in runWorkspaceDiagnostics #235 is the natural LSP-typed adapter so type errors join the delta project-wide without the file-walk.

4. Task-completion "is the project still clean?" gate. A cheap reflexive check before an agent declares done — CI-lite — currently too expensive to fire automatically.

Discipline

Value is realized through the delta, not the firehose: dumping pre-existing project errors at the agent mid-task is noise (that''s what the baseline+diff filters). So #235 earns its keep as a baseline/delta engine (capture cheap at start, diff at end), not a raw "show everything."

Load-bearing caveat — server support is the gate

Most servers do NOT advertise workspaceDiagnostics. The bulk workspace/diagnostic pull is an LSP 3.17 pull-model feature that only a minority of servers implement; many are push-only (publishDiagnostics) or support at most per-file textDocument/diagnostic. Consequences for scope:

• The per-file file-walk stays the PRIMARY path, not a fallback — bulk pull is an opportunistic optimization taken only when a server advertises diagnosticProvider.workspaceDiagnostics.
• The realized benefit is therefore per-language and partial (lands wherever a capable server runs; TypeScript is the most likely beneficiary). The error-debt-baseline and cascade-confirmation uses must degrade gracefully to runner/file-walk coverage for push-only languages — they cannot assume project-wide LSP coverage exists.
• Detection already exists (workspaceDiagnosticsSupport.mode === "pull" is necessary but NOT sufficient — that flag also covers per-file pull; need to read diagnosticProvider.workspaceDiagnostics specifically before issuing the bulk request).