Watcher can repeatedly re-index dirty repos and cause large disk write amplification

[wymfly]

Summary

We observed background watcher processes causing very large sustained disk writes during normal MCP usage. On a macOS workstation using Codex with several indexed repos, host write counters exceeded 1 TB/day.

The issue appears to be write amplification from the watcher/indexing path, not application logs.

What we observed

• Multiple processes were active for different repos.
• Some processes wrote hundreds of MB within a short sampling window while idle from the user's perspective.
• Codex SQLite TRACE/DEBUG/INFO logging was ruled out: log inserts were blocked and the log WAL stopped growing.
• Re-indexing repeated even without commits, because the worktree stayed dirty/untracked.

Suspected cause

The watcher treats a dirty/untracked worktree as changed on every poll. If the dirty state persists, each poll can trigger another re-index.

The incremental indexing path reduces some parsing work, but persistence still appears to rewrite/replace the project SQLite DB and may also rewrite the compressed artifact. For larger repos this can turn each watcher poll into hundreds of MB of writes.

This is amplified when multiple MCP clients/sessions run in the same repo or overlapping repos, since each process can independently trigger indexing and there does not appear to be a cross-process index lock/deduplication guard.

Expected behavior

The watcher should not repeatedly re-index the same unchanged dirty state.

Possible mitigations:

• Trigger background watcher re-index only on Git HEAD changes, or
• Track a dirty-state signature and re-index only when that signature changes, and
• Add a cross-process lock/deduplication guard so multiple MCP processes cannot rebuild the same repo index concurrently.

Longer term, incremental indexing should ideally avoid near-full DB/artifact rewrites when only a small set of files changed.

Related issues

This seems related to several existing reports, but combines them into one disk-write-amplification failure mode:

• watcher: continuous auto-sync re-indexing churn on Windows (real driver of #832) #841
• Incremental reindex costs ≈ a full index, especially after artifact bootstrap #867
• Add config key to disable file watcher #335
• auto_index uses tens of GB of memory in a non-git folder containing code #713
• Fast artifact export raw-copies a live WAL-mode DB — torn graph.db.zst that import then installs unchecked #895
• Installing a fresh DB file leaves the old -wal/-shm behind — SQLite replays the stale WAL over the new file #897
• Windows v0.8.1: stale graph + Permission denied on delete_project after orphan cbm process (reconfirm #277) #914

[DeusData]

Thanks for the write-amplification report. Triage: high-priority watcher/performance bug under #593.

Persistent dirty or untracked state should not cause repeated full re-index/write cycles while the user is idle. A fix should reproduce a dirty-repo polling loop and prove the watcher coalesces or suppresses repeat work without missing real file changes.

[Artemonim]

I disabled the auto-update of the MCP itself, since on the contrary it works too rarely for me while the agent is running, and I built the index rebuild into my local CI, which the agents themselves call when they finish part of the work.