Fix SQLite writer atomic cache publish

[WarGloom]

Summary

• write direct SQLite dumps to a sibling temp file and atomically publish them only after finalization succeeds
• discard temp DBs on writer errors instead of replacing the live cache with partial output
• remove destination -wal and -shm sidecars when installing the freshly rebuilt DB

Root Cause

The custom SQLite page writer opened the final cache DB with fopen(path, "wb"). In shared MCP sessions, that can truncate a live DB while SQLite readers still have the old file mapped, and it can leave old-generation WAL/SHM sidecars beside the rebuilt DB. The observed crash reached SQLite btreeInitPage during cbm_store_delete_file_hashes after the dump/reopen step.

Validation

• git diff --check
• ASAN_OPTIONS=detect_leaks=0 build/c/test-runner sqlite_writer
• ASAN_OPTIONS=detect_leaks=0 build/c/test-runner pipeline
  • passes 217 tests
  • note: existing unrelated UBSAN warning remains in src/pipeline/pass_similarity.c:152
• make -f Makefile.cbm build/c/codebase-memory-mcp
• CBM_CACHE_DIR=/tmp/cbm-wiki-mcp-pr-full-20260707-1 build/c/codebase-memory-mcp cli --index-worker index_repository '{"repo_path":"/home/nikita/work/Projects/ai/wiki-mcp"}' --response-out /tmp/cbm-wiki-mcp-pr-full-20260707-1.response
  • completed with status:"indexed", nodes:2256, edges:12934

Fixes #897.

[DeusData]

Thanks for the SQLite writer publish fix. Triage: draft high-priority cache/store integrity bug.

Review will be cautious because this touches direct DB publication: temp-file atomicity, cleanup on failure, destination WAL/SHM sidecar removal, and live-reader behavior all need explicit tests. Please also rename the PR title before it is ready for review so it describes the change directly.

[WarGloom]

Addressed.

• Renamed the PR to describe the change directly: "Fix SQLite writer atomic cache publish".
• Added explicit regression coverage for the remaining publish-integrity cases:
  • sw_publish_failure_removes_temp_output verifies a failed destination replacement discards the temporary DB output.
  • sw_publish_preserves_live_reader verifies a POSIX reader with an open transaction keeps seeing the old DB while a new connection sees the atomically published DB after replacement.
• Kept the existing coverage for pre-finalize non-truncation and destination -wal/-shm sidecar cleanup.

Local validation after rebasing onto current main:

• ASAN_OPTIONS=detect_leaks=0 build/c/test-runner sqlite_writer
• bash scripts/check-no-test-skips.sh
• git diff --check

[DeusData]

This fixes a real and nasty class — truncating the live DB under concurrent readers (#897's btreeInitPage crash) — and temp-write + atomic publish + discard-on-error is the right design. The tests and the WAL/SHM sidecar cleanup are welcome. Three changes requested:

1. Windows non-ASCII paths: the new code opens the temp file with raw fopen and publishes via MoveFileExA (ANSI). On Windows, both break for non-ASCII cache paths (e.g. a user profile with umlauts — the cache DB name derives from the repo path). Repo policy is cbm_fopen (UTF-8 → _wfopen) for any user/cache path; please use it for the temp + sidecar writes here, and use MoveFileExW with a UTF-8→wide conversion for the publish (yes, artifact.c has the same ANSI call from Fix Windows artifact refresh in write_file_atomic #492 — that's a latent bug we'd rather not duplicate; feel free to extract a shared cbm_rename_replace() helper and we'll fix artifact.c in a follow-up).

2. Durability of the publish: there's no fflush + fsync before the rename on the POSIX side (Windows at least passes MOVEFILE_WRITE_THROUGH). Without it, a power loss can atomically publish an incompletely-written file — the exact corruption the temp+rename dance is meant to prevent. fflush(fp); fsync(fileno(fp)); before fclose in the finalize path closes it.

3. Sidecar removal ordering: please confirm (and comment) that the destination -wal/-shm removal happens only on the success path relative to the rename — removing the WAL of a DB an old reader still has open is fine on POSIX (unlinked-but-open) but fails or misbehaves on Windows if a handle is held; a note on the expected ordering/failure mode would help the next reader.

With those, this is exactly the writer we want.