Commit e85c7d6
fix(index): bound + source-cap parallel retention with re-read fallback (low-RAM peak RSS)
Distilled from DeusData#685 (nguyentamdat) rebased onto current main, plus two
research-driven refinements and a genuine reproduce-first guard.
The parallel extract retains each file's source text so the fused cross-file
LSP resolve can re-parse it. That retention is transient but a peak-RSS driver.
On main the caps are flat (100 MiB/file, 2 GiB total) and a file over the cap is
silently unretained AND its cross-file resolution is skipped -- a graph-quality
gap. This change bounds retention AND keeps every cross-file edge.
- Source-text cap as a FLOOR, not just a ceiling: retention total defaults to
min(cbm_mem_budget()/8, 1 GiB), per-file min(32 MiB, total). Following the
rust-analyzer memory model, the RAM-derived default is clamped to a small
absolute ceiling so a huge-RAM host does not hold tens of GB it would re-read
cheaply. Both caps env-overridable via CBM_RETAIN_TOTAL_MB /
CBM_RETAIN_PER_FILE_MB (limits.c convention); ceilings bound only the
auto-derived default, never an explicit operator/caller choice. A dropped file
emits one index.retain_capped WARN per run.
- Bounded re-read fallback (the correctness guarantee): resolve_worker re-reads
an unretained file's source on demand (bounded, freed immediately) instead of
skipping resolution, wired at every cross-LSP site that consumes source. The
cap now only trades retained RAM for a bounded re-read, never a lost edge.
- cbm_parallel_extract_ex + opts struct (cbm_parallel_extract is now a wrapper
passing NULL -> env-derived defaults); malloc/calloc NULL-check hardening.
Reproduce-first: parallel_cross_file_reread_preserves_unretained_edges uses a
Java<->Kotlin pair whose cross-file lsp edges are genuinely source-dependent;
the edges are lost when the caller is unretained and the fallback is absent
(RED), present with it (GREEN), with a retained CONTROL scenario proving
non-vacuity. DeusData#685's original Python red test was a false guard (per-file py_lsp
already resolves those calls) and is replaced. Peak-bound guards
(retained_bytes <= total_cap; retain_sources=false retains nothing) in test_mem.c.
Verify: make -f Makefile.cbm cbm && make -f Makefile.cbm lint-ci; test-runner
parallel pipeline incremental py_lsp ts_lsp java_lsp kotlin_lsp c_lsp cs_lsp
go_lsp rust_lsp mem -> 2323 passed.
Co-authored-by: nguyentamdat <nguyentamdat@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>1 parent 54efcef commit e85c7d6
4 files changed
Lines changed: 552 additions & 69 deletions
0 commit comments