feat(minidb): persistent index generations and lifecycle hardening - #2604
Merged
Conversation
|
- everysec WAL now fsyncs on the timer only while dirty (tracked by a write/sync generation watermark); close() keeps its unconditional final sync, and background sync failures surface via walFsyncErrors plus a sticky lastWalFsyncError instead of being silently swallowed - add WAL queue/group-commit counters (walQueuedBytes, walMaxQueuedBytes, walGroupCommits, walGroupCommitFrames) and lifecycle phase stats: recovery bytes/frames/duration, index/text rebuild durations, compaction total/snapshot/rotation/postings durations, rotation pause, and query candidates/decoded/sorted rows; add a syncIntervalMs open option threaded through compaction WAL rotation - rewrite the bench on fixed-seed synthetic data with a stable machine-readable JSON report (cold open 10k/50k/100k, word/ngram search, idle-fsync acceptance, 100k compaction, event-loop delay, peak heap/RSS per scenario) and pin the schema in test/bench-json.test.ts; add app-side baselines with loose complexity budgets in sessionIndex and searchService tests - fix ClusterDb lock-pool closeAll() leaking in-flight shard opens and drain the query store's async close on server shutdown, eliminating the ENOTEMPTY directory-teardown race
- rebuild all derived indexes in one shared store walk: a single decode per record fans out to staged builders, dt rebuild reads record metadata only, and index-less opens no longer decode at all - rank full-text results with a bounded min-heap plus a stable key tie-break instead of sorting every candidate - remove/overwrite text docs via a docID -> delta-terms reverse map instead of scanning the whole delta vocabulary - validate unique batches incrementally against touched postings instead of copying the full per-index owner map - reap due TTL entries from the expiry heap on the write path instead of a full-store sweep per write
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e5dd9a299a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+452
to
+454
| if (workerTargets.size > 0) { | ||
| for (const [, { ti }] of workerTargets) ti.beginRebase(); | ||
| await gb.wal.flush(); |
There was a problem hiding this comment.
When workerTargets.size === 0 (for example a database with only secondary/dt indexes, clean text indexes, or small/custom text indexes), this branch skips gb.wal.flush() before the generation is written and published. The generation image has already drained queued mutations from in-flight writes, but those writes may still be sitting in the WAL queue and can later fail/poison and roll back in memory; the published generation would then contain a write the caller saw rejected, and a future open can resurrect it. Please make the WAL flush/check part unconditional before publishing the checkpoint, not only a prerequisite for worker text builds.
Useful? React with 👍 / 👎.
Comment on lines
+440
to
+442
| if (page.nextCursor === undefined) break; | ||
| before = page.nextCursor; | ||
| after = undefined; |
There was a problem hiding this comment.
For GET /sessions requests that start with after_id, this drain loop switches subsequent reads to before = page.nextCursor and clears after, so if the first bounded index pages are filtered out by exclude_empty, missing cwd, or the archived-only/busy filters, the loop can continue past the original cursor and return sessions at or older than after_id. The previous full-list slicing kept the newer-than-cursor window fixed; this path needs to keep that lower bound while fetching more candidates or avoid draining across it.
Useful? React with 👍 / 👎.
sailist
force-pushed
the
feat/search-index-hardening
branch
from
August 4, 2026 11:42
e5dd9a2 to
570c219
Compare
commit: |
- add ISessionIndex read-model lifecycle (prepare/status, ready/degraded states) behind the persistence_minidb_readmodel experimental flag - add ISessionIndexMirror write side recording fresh summaries into a bounded, coalescing queue after the authoritative document is durable - replace the offset cursor with before/after keyset pagination; rename list/countActive to listRecent/count - extend IQueryStore with ordered columns and pageByColumn, plus getMany/listKeys/dropCollection - wire the read model through kap-server routes and start, and update the klient sessions contract - index every session for global search instead of the 500 most recent
…budgets - split search requests from sync work: searchIndex() no longer awaits runSync/reopen/reindex; a single-flight sync coordinator with debounce and backpressure runs in the background, stale generations keep serving with explicit stale/degraded state, and refresh/sync/reindex failures surface via lastRefreshError instead of being swallowed - scope file-meta keys by session id (\0meta\file\<sessionId>\<hash>) with lazy + one-shot background migration from the legacy hash-only keys, so one session sync only touches its own meta rows - make authoritative scans incremental (mtime/ino/size rescan conditions, unchanged files no longer rewrite meta) and read wire deltas in 1 MiB chunks instead of whole-file buffer + split - replace offset pagination with versioned v2 keyset page tokens (fingerprint + index generation + sort boundary); generation changes fail old tokens with invalid_page_token, legacy v1 offset tokens are served once and upgraded, and pages collect via bounded top-K instead of full sort + offset skip - add query budgets enforced at the postings/score stage: max query terms, literal length cap, postings visit budget (minidb searchBounded/maxVisits with prefix decoding that never fabricates hits and skips the postings LRU), candidate caps, deadline and text budget; truncation is reported via incomplete reasons candidate_cap/postings_budget/deadline - reopen read-only dbs by opening the next handle before closing the previous one so a failed refresh keeps the old generation serving; failed opens now self-heal through search traffic 100k-message bench: first page p95 < 300ms and page-100 cost on par with page 1; event-loop delay during queries stays sub-millisecond.
- give WAL writes a commit point: a failed flushBatch poisons the WAL (WAL_POISONED, tracked separately as walWriteErrors vs walFsyncErrors), rejects queued frames in reverse enqueue order, and stops scheduling further batches; everysec background sync failures stay non-rejecting per stage-1 semantics - recover in place to a known-safe point: a serialized recovery chain truncates the WAL back to the first un-acked frame, rebuilds size/nextOffset, and clears the poison; writes queue behind the recovery gate (zero-cost when idle), a failed truncate flips the instance into an explicit writeDisabled state, and a stale truncate offset (WAL file replaced by a rotation) skips the truncate - roll failed flush groups back as a unit: frames are stamped with their batchId, MiniDb keeps per-group earliest pre-state, and the first rejection restores every key of the group (rejected writes no longer reappear after reopen, and in-memory state matches reopen for any failure interleaving); the per-op seq guard remains for cross-group and rotation-retry races - wrap applyOp and the following in-memory mutations so a contract violation poisons the WAL and rolls the group back instead of escaping as a half-commit; frames never enqueued (seal race) roll back per-op without poisoning - tag errors past the commit point with ambiguous: true so callers can distinguish "definitely not applied" from "maybe applied but revoked" - close() waits for the recovery chain to go idle and backup() fences behind in-flight recovery before copying files Controlled A/B bench (22 alternating iterations, 100k concurrent sets): write-path throughput regression is within the 2% budget.
- distinguish lock ownership by instance instead of pid: every acquire mints a pid:uuid token carried by lock/bid/watch files, inspect().mine compares tokens, liveness still follows pid, tokenless legacy files keep the old stale-takeover path, and hasLiveForeignWatch excludes self by token so same-process contenders see each other (closing the double-win takeover and the cross-instance release); a live same-pid lock is still respected, and re-acquiring a held lock is idempotent - serialize acquire/renew/release through a per-instance promise-chain mutex: renew re-checks held inside the chain and release waits for an in-flight renew, eliminating the renew/rename-after-unlink ghost lock - make MiniDb.close() a state machine (open/closing/closed) with a shared closePromise: cleanup runs per-resource try/catch in dependency order (text indexes, store, valueReader, WAL, lock), aggregates every cleanup error into an AggregateError, stays in 'closing' on failure so a retry finishes the cleanup, and no longer leaks the lock when the WAL close fails; a rejected in-flight compaction no longer escapes the cleanup pass
- add an internal persistent-files module as the single source of truth for the persisted file set (snapshot, WAL, sidecars, postings pattern, fingerprint subset); lock-pool fingerprints, persistentFiles, open stale-tmp cleanup, and backup/restore filtering all derive from it, and fingerprints upgrade to dev:ino:size:mtimeMs so compound sidecar changes can no longer hide from cluster readers - pair snapshot and WAL generations during recovery (transitional stat-pairing until stage-5 manifests): each pass anchors the fds it scans, re-stats afterwards, tolerates append-only WAL growth, retries bounded times on generation churn with a clean store reset, and throws RECOVERY_GENERATION_CHURN when churn exceeds the budget; the disk-mode ValueReader attach re-validates inodes so stale offsets never read a replaced file - make the rotation directory fsyncs strict: failures abort the rotation through the existing rollback path instead of being swallowed, while platforms without directory fsync degrade once with a warn and stats.dirFsyncUnsupported
…fore publish - extract the promise-chain mutex into a shared createSerializer() and give each sidecar family (secondary/compound/text) its own chain: create/drop run uninterruptibly (memory change + rebuild + persist), different families stay independent, and the data write path never shares these chains - reverse the publication order to staged -> persist -> publish: a create stages the definition, rebuilds via the staged builder, persists the sidecar including the new definition, then publishes atomically; any failure discards the staged state leaving live and sidecar untouched (no phantom indexes, retry-safe); a drop persists the sidecar without the definition before removing it live; text index create/drop adopt the same pattern, replacing the hand-rolled unwind, and a dropping marker keeps compaction postings rebuilds out of the persist window - feed staged indexes from the incremental write path (add/remove/ checkUnique/checkUniqueBatch visit live+staged) so writes landing in the persist window are not lost at publish; queries still see live only - harden writeFileAtomic: instance-unique tmp names (.tmp-pid-seq), a strict fsyncDir after rename so a successful persist is crash durable, and whitelist-based stale-tmp cleanup that never touches lock tmp files
…ues once - canonical value at the write boundary: the json codec re-parses the encoded bytes once and every downstream consumer (unique checks, secondary/compound/text indexes, dt extraction) sees exactly the persisted representation, so getter/toJSON/Proxy documents can no longer diverge between the index view and the storage view - reorder the set/batch pipeline so every fallible check happens before any visible side effect: prepare (key/ttl checks, encoding, canonical decode, index field extraction, tokenization) -> unique checks -> ensureMemoryFor eviction -> commit; a constraint failure now leaves the database untouched (no more evicted victims on rejected inserts), and applyOp is structurally pure against pre-validated data - tokenize at the prepare boundary: TextIndex gains prepareAdd/ addPrepared and the buildQueue carries validated key+tokens mutations instead of raw docs, so a throwing custom tokenizer can no longer poison the live view or the queue, and custom-tokenizer output is rejected per token over 0xffff bytes before it can permanently break postings rebuilds; prepared tokens are keyed by index instance so a same-name drop+create mid-write re-tokenizes instead of crossing tokenizers - strict batch structure validation: scanBatchOpRefs/decodeBatchOps reject unknown op types, out-of-bounds lengths, and trailing bytes (offset must equal body length), so a valid-CRC but malformed batch is skipped as a unit and counted via RecoveryInfo.corruptBatches instead of being partially applied Bench vs the stage-1 baseline: json write throughput regression is within the 5% budget (median ~2-4% depending on the measurement).
… tests - introduce the internal OpTracker (close gate + in-flight counter with enter/leave/close/whenIdle and reference-counted pause/resume) and drive every shutdown/drain path from it: WAL background syncs are tracked so close() waits out an in-flight sync before closing the fd, cluster lock-pool closeAll() closes the gates and drains busy callbacks before closing handles, and MiniDb writes pass a write gate - make backup() atomic with a defined linearization point: pause the write gate, drain in-flight writes (every acknowledged write is now included), copy to a sibling temp dir with per-file fsyncs, write the manifest last as the commit marker, and rename into place; failures clean up and leave no partial backup, and concurrent writes are rejected with BACKUP_IN_PROGRESS - reap emptied compound-index groups on remove (the groups map no longer grows monotonically), move the open-time mkdir behind the readOnly check so a read-only open of a missing directory fails with ENOENT instead of creating it, and never run a destructive rebuild for a read-only open failure (explicit or onLockFail fallback) - consolidate every review fault-injection repro into the formal suite behind deterministic barrier helpers (programmable writev/sync/ rename/tokenize hooks) and convert the six timing-based tests to barrier/tick-driven assertions; the .tmp repro scripts are removed The converted timing tests and the full suite pass 50 repeat runs (including under CPU load injection) with zero flakes.
…m WAL delta - checkpoint the store, dt/secondary/compound indexes, and text dictionary/postings/docs into immutable generations under generations/g-NNNNNN published atomically (tmp build, per-file checksums and fsyncs, dir rename, CURRENT swap, strict dir fsyncs); the manifest records the format version, WAL/snapshot checkpoint anchors, per-index definition hashes, and codec/value-mode compatibility - open now loads the published generation and replays only the WAL delta after its checkpoint: no full value decode, corpus tokenization, or postings rewrite on a normal reopen (warm opens are 3.5-13.8x faster at 100k/1M records); a definition change rebuilds only the affected index, and corrupt generation files fall back to the previous generation or the legacy full recovery without ever touching the authoritative snapshot/WAL - build generations transactionally with compaction (rotation plus derived state publish as one unit, replacing the synchronous rebuildTextPostings tail), capture concurrent writes through a sealed op queue with byte/op caps, hard-link clean postings and the snapshot into the new generation, and repoint every live text base into the CURRENT generation after publish - cluster/read-only refresh watches CURRENT and the WAL watermark: pure generation publishes keep readers on incremental catch-up while rotations reopen onto the new generation; writers building the next generation never disturb readers of the current one - legacy databases open through the old path unchanged and gain their first generation in the background; OpenOptions.indexGenerations: false fully restores the pre-generation behavior
- split the monolithic src/index.ts into facet modules (mini-db, types, value-codec, memory-guard, backup, query-engine, text-registry, wal-group, generation-builder/loader, write-path, read-path, index-admin, lifecycle, stats) and move text-index.ts to text-index/ - run corpus-scale text-index builds off the main thread via the bounded worker engine (src/worker/), exported through the new worker-runtime subpath, with inline fallback for small corpora and rollback switches - defer the open-time fallback text rebuild into a maintenance task; searches on a not-yet-committed base raise TextIndexBuildingError - add the unified maintenance scheduler, bounded async read surface, and a maintenance bench - kap-server search: switch to searchBoundedAsync and serve the building page while the index base rebuilds after fallback recovery - kimi-code: install the SEA-bundled minidb text-build worker at startup, bundle it via the native asset scripts, and add the startup-trace util plus the KIMI_TUI_INPUT_LATENCY debug probe
- extract isUnsupportedDirectoryFsyncError and cover win32 EPERM - drop the one-shot console.warn; stats.dirFsyncUnsupported carries the degraded state
- dispose() now closes an OpTracker gate and drains in-flight sync/refresh passes before closing the db, so no background write can hit a closed handle; the deleteSessionDocs loop and trailing stats write skip once the gate closes (review MoonshotAI#20) - drainGlobalSearchDisposals loops to a fixpoint so disposals registered while a drain is in flight are also awaited (review MoonshotAI#21) - pin the post-open failure semantics with a regression test: a failed text-index setup closes the handle and the next open reacquires the writer lock instead of self-locking read-only (review MoonshotAI#19) - export OpTracker from the minidb root for the search service's drain
sailist
force-pushed
the
feat/search-index-hardening
branch
from
August 4, 2026 12:05
570c219 to
373412e
Compare
mbuckaway
pushed a commit
to mbuckaway/kimi-code
that referenced
this pull request
Aug 4, 2026
…oonshotAI#2604) * perf(minidb): skip idle everysec fsyncs and add lifecycle stats - everysec WAL now fsyncs on the timer only while dirty (tracked by a write/sync generation watermark); close() keeps its unconditional final sync, and background sync failures surface via walFsyncErrors plus a sticky lastWalFsyncError instead of being silently swallowed - add WAL queue/group-commit counters (walQueuedBytes, walMaxQueuedBytes, walGroupCommits, walGroupCommitFrames) and lifecycle phase stats: recovery bytes/frames/duration, index/text rebuild durations, compaction total/snapshot/rotation/postings durations, rotation pause, and query candidates/decoded/sorted rows; add a syncIntervalMs open option threaded through compaction WAL rotation - rewrite the bench on fixed-seed synthetic data with a stable machine-readable JSON report (cold open 10k/50k/100k, word/ngram search, idle-fsync acceptance, 100k compaction, event-loop delay, peak heap/RSS per scenario) and pin the schema in test/bench-json.test.ts; add app-side baselines with loose complexity budgets in sessionIndex and searchService tests - fix ClusterDb lock-pool closeAll() leaking in-flight shard opens and drain the query store's async close on server shutdown, eliminating the ENOTEMPTY directory-teardown race * perf(minidb): bound startup rebuild and steady-state hot paths - rebuild all derived indexes in one shared store walk: a single decode per record fans out to staged builders, dt rebuild reads record metadata only, and index-less opens no longer decode at all - rank full-text results with a bounded min-heap plus a stable key tie-break instead of sorting every candidate - remove/overwrite text docs via a docID -> delta-terms reverse map instead of scanning the whole delta vocabulary - validate unique batches incrementally against touched postings instead of copying the full per-index owner map - reap due TTL entries from the expiry heap on the write path instead of a full-store sweep per write * feat(session-index): add minidb read model with keyset pagination - add ISessionIndex read-model lifecycle (prepare/status, ready/degraded states) behind the persistence_minidb_readmodel experimental flag - add ISessionIndexMirror write side recording fresh summaries into a bounded, coalescing queue after the authoritative document is durable - replace the offset cursor with before/after keyset pagination; rename list/countActive to listRecent/count - extend IQueryStore with ordered columns and pageByColumn, plus getMany/listKeys/dropCollection - wire the read model through kap-server routes and start, and update the klient sessions contract - index every session for global search instead of the 500 most recent * feat(kap-server): bound search sync lifecycle, pagination, and query budgets - split search requests from sync work: searchIndex() no longer awaits runSync/reopen/reindex; a single-flight sync coordinator with debounce and backpressure runs in the background, stale generations keep serving with explicit stale/degraded state, and refresh/sync/reindex failures surface via lastRefreshError instead of being swallowed - scope file-meta keys by session id (\0meta\file\<sessionId>\<hash>) with lazy + one-shot background migration from the legacy hash-only keys, so one session sync only touches its own meta rows - make authoritative scans incremental (mtime/ino/size rescan conditions, unchanged files no longer rewrite meta) and read wire deltas in 1 MiB chunks instead of whole-file buffer + split - replace offset pagination with versioned v2 keyset page tokens (fingerprint + index generation + sort boundary); generation changes fail old tokens with invalid_page_token, legacy v1 offset tokens are served once and upgraded, and pages collect via bounded top-K instead of full sort + offset skip - add query budgets enforced at the postings/score stage: max query terms, literal length cap, postings visit budget (minidb searchBounded/maxVisits with prefix decoding that never fabricates hits and skips the postings LRU), candidate caps, deadline and text budget; truncation is reported via incomplete reasons candidate_cap/postings_budget/deadline - reopen read-only dbs by opening the next handle before closing the previous one so a failed refresh keeps the old generation serving; failed opens now self-heal through search traffic 100k-message bench: first page p95 < 300ms and page-100 cost on par with page 1; event-loop delay during queries stays sub-millisecond. * fix(minidb): poison, roll back, and recover the WAL on write failures - give WAL writes a commit point: a failed flushBatch poisons the WAL (WAL_POISONED, tracked separately as walWriteErrors vs walFsyncErrors), rejects queued frames in reverse enqueue order, and stops scheduling further batches; everysec background sync failures stay non-rejecting per stage-1 semantics - recover in place to a known-safe point: a serialized recovery chain truncates the WAL back to the first un-acked frame, rebuilds size/nextOffset, and clears the poison; writes queue behind the recovery gate (zero-cost when idle), a failed truncate flips the instance into an explicit writeDisabled state, and a stale truncate offset (WAL file replaced by a rotation) skips the truncate - roll failed flush groups back as a unit: frames are stamped with their batchId, MiniDb keeps per-group earliest pre-state, and the first rejection restores every key of the group (rejected writes no longer reappear after reopen, and in-memory state matches reopen for any failure interleaving); the per-op seq guard remains for cross-group and rotation-retry races - wrap applyOp and the following in-memory mutations so a contract violation poisons the WAL and rolls the group back instead of escaping as a half-commit; frames never enqueued (seal race) roll back per-op without poisoning - tag errors past the commit point with ambiguous: true so callers can distinguish "definitely not applied" from "maybe applied but revoked" - close() waits for the recovery chain to go idle and backup() fences behind in-flight recovery before copying files Controlled A/B bench (22 alternating iterations, 100k concurrent sets): write-path throughput regression is within the 2% budget. * fix(minidb): turn the file lock into an instance-owned serialized lease - distinguish lock ownership by instance instead of pid: every acquire mints a pid:uuid token carried by lock/bid/watch files, inspect().mine compares tokens, liveness still follows pid, tokenless legacy files keep the old stale-takeover path, and hasLiveForeignWatch excludes self by token so same-process contenders see each other (closing the double-win takeover and the cross-instance release); a live same-pid lock is still respected, and re-acquiring a held lock is idempotent - serialize acquire/renew/release through a per-instance promise-chain mutex: renew re-checks held inside the chain and release waits for an in-flight renew, eliminating the renew/rename-after-unlink ghost lock - make MiniDb.close() a state machine (open/closing/closed) with a shared closePromise: cleanup runs per-resource try/catch in dependency order (text indexes, store, valueReader, WAL, lock), aggregates every cleanup error into an AggregateError, stays in 'closing' on failure so a retry finishes the cleanup, and no longer leaks the lock when the WAL close fails; a rejected in-flight compaction no longer escapes the cleanup pass * fix(minidb): keep readers on one consistent file generation - add an internal persistent-files module as the single source of truth for the persisted file set (snapshot, WAL, sidecars, postings pattern, fingerprint subset); lock-pool fingerprints, persistentFiles, open stale-tmp cleanup, and backup/restore filtering all derive from it, and fingerprints upgrade to dev:ino:size:mtimeMs so compound sidecar changes can no longer hide from cluster readers - pair snapshot and WAL generations during recovery (transitional stat-pairing until stage-5 manifests): each pass anchors the fds it scans, re-stats afterwards, tolerates append-only WAL growth, retries bounded times on generation churn with a clean store reset, and throws RECOVERY_GENERATION_CHURN when churn exceeds the budget; the disk-mode ValueReader attach re-validates inodes so stale offsets never read a replaced file - make the rotation directory fsyncs strict: failures abort the rotation through the existing rollback path instead of being swallowed, while platforms without directory fsync degrade once with a warn and stats.dirFsyncUnsupported * fix(minidb): serialize index-definition sidecar mutations, persist before publish - extract the promise-chain mutex into a shared createSerializer() and give each sidecar family (secondary/compound/text) its own chain: create/drop run uninterruptibly (memory change + rebuild + persist), different families stay independent, and the data write path never shares these chains - reverse the publication order to staged -> persist -> publish: a create stages the definition, rebuilds via the staged builder, persists the sidecar including the new definition, then publishes atomically; any failure discards the staged state leaving live and sidecar untouched (no phantom indexes, retry-safe); a drop persists the sidecar without the definition before removing it live; text index create/drop adopt the same pattern, replacing the hand-rolled unwind, and a dropping marker keeps compaction postings rebuilds out of the persist window - feed staged indexes from the incremental write path (add/remove/ checkUnique/checkUniqueBatch visit live+staged) so writes landing in the persist window are not lost at publish; queries still see live only - harden writeFileAtomic: instance-unique tmp names (.tmp-pid-seq), a strict fsyncDir after rename so a successful persist is crash durable, and whitelist-based stale-tmp cleanup that never touches lock tmp files * fix(minidb): validate writes before any side effect, canonicalize values once - canonical value at the write boundary: the json codec re-parses the encoded bytes once and every downstream consumer (unique checks, secondary/compound/text indexes, dt extraction) sees exactly the persisted representation, so getter/toJSON/Proxy documents can no longer diverge between the index view and the storage view - reorder the set/batch pipeline so every fallible check happens before any visible side effect: prepare (key/ttl checks, encoding, canonical decode, index field extraction, tokenization) -> unique checks -> ensureMemoryFor eviction -> commit; a constraint failure now leaves the database untouched (no more evicted victims on rejected inserts), and applyOp is structurally pure against pre-validated data - tokenize at the prepare boundary: TextIndex gains prepareAdd/ addPrepared and the buildQueue carries validated key+tokens mutations instead of raw docs, so a throwing custom tokenizer can no longer poison the live view or the queue, and custom-tokenizer output is rejected per token over 0xffff bytes before it can permanently break postings rebuilds; prepared tokens are keyed by index instance so a same-name drop+create mid-write re-tokenizes instead of crossing tokenizers - strict batch structure validation: scanBatchOpRefs/decodeBatchOps reject unknown op types, out-of-bounds lengths, and trailing bytes (offset must equal body length), so a valid-CRC but malformed batch is skipped as a unit and counted via RecoveryInfo.corruptBatches instead of being partially applied Bench vs the stage-1 baseline: json write throughput regression is within the 5% budget (median ~2-4% depending on the measurement). * feat(minidb): add OpTracker drain primitive and atomic backup, harden tests - introduce the internal OpTracker (close gate + in-flight counter with enter/leave/close/whenIdle and reference-counted pause/resume) and drive every shutdown/drain path from it: WAL background syncs are tracked so close() waits out an in-flight sync before closing the fd, cluster lock-pool closeAll() closes the gates and drains busy callbacks before closing handles, and MiniDb writes pass a write gate - make backup() atomic with a defined linearization point: pause the write gate, drain in-flight writes (every acknowledged write is now included), copy to a sibling temp dir with per-file fsyncs, write the manifest last as the commit marker, and rename into place; failures clean up and leave no partial backup, and concurrent writes are rejected with BACKUP_IN_PROGRESS - reap emptied compound-index groups on remove (the groups map no longer grows monotonically), move the open-time mkdir behind the readOnly check so a read-only open of a missing directory fails with ENOENT instead of creating it, and never run a destructive rebuild for a read-only open failure (explicit or onLockFail fallback) - consolidate every review fault-injection repro into the formal suite behind deterministic barrier helpers (programmable writev/sync/ rename/tokenize hooks) and convert the six timing-based tests to barrier/tick-driven assertions; the .tmp repro scripts are removed The converted timing tests and the full suite pass 50 repeat runs (including under CPU load injection) with zero flakes. * feat(minidb): persist derived indexes as atomic generations, open from WAL delta - checkpoint the store, dt/secondary/compound indexes, and text dictionary/postings/docs into immutable generations under generations/g-NNNNNN published atomically (tmp build, per-file checksums and fsyncs, dir rename, CURRENT swap, strict dir fsyncs); the manifest records the format version, WAL/snapshot checkpoint anchors, per-index definition hashes, and codec/value-mode compatibility - open now loads the published generation and replays only the WAL delta after its checkpoint: no full value decode, corpus tokenization, or postings rewrite on a normal reopen (warm opens are 3.5-13.8x faster at 100k/1M records); a definition change rebuilds only the affected index, and corrupt generation files fall back to the previous generation or the legacy full recovery without ever touching the authoritative snapshot/WAL - build generations transactionally with compaction (rotation plus derived state publish as one unit, replacing the synchronous rebuildTextPostings tail), capture concurrent writes through a sealed op queue with byte/op caps, hard-link clean postings and the snapshot into the new generation, and repoint every live text base into the CURRENT generation after publish - cluster/read-only refresh watches CURRENT and the WAL watermark: pure generation publishes keep readers on incremental catch-up while rotations reopen onto the new generation; writers building the next generation never disturb readers of the current one - legacy databases open through the old path unchanged and gain their first generation in the background; OpenOptions.indexGenerations: false fully restores the pre-generation behavior * feat(minidb): workerize text-index builds and split MiniDb into facets - split the monolithic src/index.ts into facet modules (mini-db, types, value-codec, memory-guard, backup, query-engine, text-registry, wal-group, generation-builder/loader, write-path, read-path, index-admin, lifecycle, stats) and move text-index.ts to text-index/ - run corpus-scale text-index builds off the main thread via the bounded worker engine (src/worker/), exported through the new worker-runtime subpath, with inline fallback for small corpora and rollback switches - defer the open-time fallback text rebuild into a maintenance task; searches on a not-yet-committed base raise TextIndexBuildingError - add the unified maintenance scheduler, bounded async read surface, and a maintenance bench - kap-server search: switch to searchBoundedAsync and serve the building page while the index base rebuilds after fallback recovery - kimi-code: install the SEA-bundled minidb text-build worker at startup, bundle it via the native asset scripts, and add the startup-trace util plus the KIMI_TUI_INPUT_LATENCY debug probe * fix(minidb): treat win32 EPERM as unsupported directory fsync - extract isUnsupportedDirectoryFsyncError and cover win32 EPERM - drop the one-shot console.warn; stats.dirFsyncUnsupported carries the degraded state * fix(kap-server): harden search-index dispose and drain lifecycle - dispose() now closes an OpTracker gate and drains in-flight sync/refresh passes before closing the db, so no background write can hit a closed handle; the deleteSessionDocs loop and trailing stats write skip once the gate closes (review MoonshotAI#20) - drainGlobalSearchDisposals loops to a fixpoint so disposals registered while a drain is in flight are also awaited (review MoonshotAI#21) - pin the post-open failure semantics with a regression test: a failed text-index setup closes the handle and the next open reacquires the writer lock instead of self-locking read-only (review MoonshotAI#19) - export OpTracker from the minidb root for the search service's drain * chore: fix oxlint type-aware lint errors
daofazhiran
pushed a commit
to daofazhiran/kimi-code
that referenced
this pull request
Aug 4, 2026
…oonshotAI#2604) * perf(minidb): skip idle everysec fsyncs and add lifecycle stats - everysec WAL now fsyncs on the timer only while dirty (tracked by a write/sync generation watermark); close() keeps its unconditional final sync, and background sync failures surface via walFsyncErrors plus a sticky lastWalFsyncError instead of being silently swallowed - add WAL queue/group-commit counters (walQueuedBytes, walMaxQueuedBytes, walGroupCommits, walGroupCommitFrames) and lifecycle phase stats: recovery bytes/frames/duration, index/text rebuild durations, compaction total/snapshot/rotation/postings durations, rotation pause, and query candidates/decoded/sorted rows; add a syncIntervalMs open option threaded through compaction WAL rotation - rewrite the bench on fixed-seed synthetic data with a stable machine-readable JSON report (cold open 10k/50k/100k, word/ngram search, idle-fsync acceptance, 100k compaction, event-loop delay, peak heap/RSS per scenario) and pin the schema in test/bench-json.test.ts; add app-side baselines with loose complexity budgets in sessionIndex and searchService tests - fix ClusterDb lock-pool closeAll() leaking in-flight shard opens and drain the query store's async close on server shutdown, eliminating the ENOTEMPTY directory-teardown race * perf(minidb): bound startup rebuild and steady-state hot paths - rebuild all derived indexes in one shared store walk: a single decode per record fans out to staged builders, dt rebuild reads record metadata only, and index-less opens no longer decode at all - rank full-text results with a bounded min-heap plus a stable key tie-break instead of sorting every candidate - remove/overwrite text docs via a docID -> delta-terms reverse map instead of scanning the whole delta vocabulary - validate unique batches incrementally against touched postings instead of copying the full per-index owner map - reap due TTL entries from the expiry heap on the write path instead of a full-store sweep per write * feat(session-index): add minidb read model with keyset pagination - add ISessionIndex read-model lifecycle (prepare/status, ready/degraded states) behind the persistence_minidb_readmodel experimental flag - add ISessionIndexMirror write side recording fresh summaries into a bounded, coalescing queue after the authoritative document is durable - replace the offset cursor with before/after keyset pagination; rename list/countActive to listRecent/count - extend IQueryStore with ordered columns and pageByColumn, plus getMany/listKeys/dropCollection - wire the read model through kap-server routes and start, and update the klient sessions contract - index every session for global search instead of the 500 most recent * feat(kap-server): bound search sync lifecycle, pagination, and query budgets - split search requests from sync work: searchIndex() no longer awaits runSync/reopen/reindex; a single-flight sync coordinator with debounce and backpressure runs in the background, stale generations keep serving with explicit stale/degraded state, and refresh/sync/reindex failures surface via lastRefreshError instead of being swallowed - scope file-meta keys by session id (\0meta\file\<sessionId>\<hash>) with lazy + one-shot background migration from the legacy hash-only keys, so one session sync only touches its own meta rows - make authoritative scans incremental (mtime/ino/size rescan conditions, unchanged files no longer rewrite meta) and read wire deltas in 1 MiB chunks instead of whole-file buffer + split - replace offset pagination with versioned v2 keyset page tokens (fingerprint + index generation + sort boundary); generation changes fail old tokens with invalid_page_token, legacy v1 offset tokens are served once and upgraded, and pages collect via bounded top-K instead of full sort + offset skip - add query budgets enforced at the postings/score stage: max query terms, literal length cap, postings visit budget (minidb searchBounded/maxVisits with prefix decoding that never fabricates hits and skips the postings LRU), candidate caps, deadline and text budget; truncation is reported via incomplete reasons candidate_cap/postings_budget/deadline - reopen read-only dbs by opening the next handle before closing the previous one so a failed refresh keeps the old generation serving; failed opens now self-heal through search traffic 100k-message bench: first page p95 < 300ms and page-100 cost on par with page 1; event-loop delay during queries stays sub-millisecond. * fix(minidb): poison, roll back, and recover the WAL on write failures - give WAL writes a commit point: a failed flushBatch poisons the WAL (WAL_POISONED, tracked separately as walWriteErrors vs walFsyncErrors), rejects queued frames in reverse enqueue order, and stops scheduling further batches; everysec background sync failures stay non-rejecting per stage-1 semantics - recover in place to a known-safe point: a serialized recovery chain truncates the WAL back to the first un-acked frame, rebuilds size/nextOffset, and clears the poison; writes queue behind the recovery gate (zero-cost when idle), a failed truncate flips the instance into an explicit writeDisabled state, and a stale truncate offset (WAL file replaced by a rotation) skips the truncate - roll failed flush groups back as a unit: frames are stamped with their batchId, MiniDb keeps per-group earliest pre-state, and the first rejection restores every key of the group (rejected writes no longer reappear after reopen, and in-memory state matches reopen for any failure interleaving); the per-op seq guard remains for cross-group and rotation-retry races - wrap applyOp and the following in-memory mutations so a contract violation poisons the WAL and rolls the group back instead of escaping as a half-commit; frames never enqueued (seal race) roll back per-op without poisoning - tag errors past the commit point with ambiguous: true so callers can distinguish "definitely not applied" from "maybe applied but revoked" - close() waits for the recovery chain to go idle and backup() fences behind in-flight recovery before copying files Controlled A/B bench (22 alternating iterations, 100k concurrent sets): write-path throughput regression is within the 2% budget. * fix(minidb): turn the file lock into an instance-owned serialized lease - distinguish lock ownership by instance instead of pid: every acquire mints a pid:uuid token carried by lock/bid/watch files, inspect().mine compares tokens, liveness still follows pid, tokenless legacy files keep the old stale-takeover path, and hasLiveForeignWatch excludes self by token so same-process contenders see each other (closing the double-win takeover and the cross-instance release); a live same-pid lock is still respected, and re-acquiring a held lock is idempotent - serialize acquire/renew/release through a per-instance promise-chain mutex: renew re-checks held inside the chain and release waits for an in-flight renew, eliminating the renew/rename-after-unlink ghost lock - make MiniDb.close() a state machine (open/closing/closed) with a shared closePromise: cleanup runs per-resource try/catch in dependency order (text indexes, store, valueReader, WAL, lock), aggregates every cleanup error into an AggregateError, stays in 'closing' on failure so a retry finishes the cleanup, and no longer leaks the lock when the WAL close fails; a rejected in-flight compaction no longer escapes the cleanup pass * fix(minidb): keep readers on one consistent file generation - add an internal persistent-files module as the single source of truth for the persisted file set (snapshot, WAL, sidecars, postings pattern, fingerprint subset); lock-pool fingerprints, persistentFiles, open stale-tmp cleanup, and backup/restore filtering all derive from it, and fingerprints upgrade to dev:ino:size:mtimeMs so compound sidecar changes can no longer hide from cluster readers - pair snapshot and WAL generations during recovery (transitional stat-pairing until stage-5 manifests): each pass anchors the fds it scans, re-stats afterwards, tolerates append-only WAL growth, retries bounded times on generation churn with a clean store reset, and throws RECOVERY_GENERATION_CHURN when churn exceeds the budget; the disk-mode ValueReader attach re-validates inodes so stale offsets never read a replaced file - make the rotation directory fsyncs strict: failures abort the rotation through the existing rollback path instead of being swallowed, while platforms without directory fsync degrade once with a warn and stats.dirFsyncUnsupported * fix(minidb): serialize index-definition sidecar mutations, persist before publish - extract the promise-chain mutex into a shared createSerializer() and give each sidecar family (secondary/compound/text) its own chain: create/drop run uninterruptibly (memory change + rebuild + persist), different families stay independent, and the data write path never shares these chains - reverse the publication order to staged -> persist -> publish: a create stages the definition, rebuilds via the staged builder, persists the sidecar including the new definition, then publishes atomically; any failure discards the staged state leaving live and sidecar untouched (no phantom indexes, retry-safe); a drop persists the sidecar without the definition before removing it live; text index create/drop adopt the same pattern, replacing the hand-rolled unwind, and a dropping marker keeps compaction postings rebuilds out of the persist window - feed staged indexes from the incremental write path (add/remove/ checkUnique/checkUniqueBatch visit live+staged) so writes landing in the persist window are not lost at publish; queries still see live only - harden writeFileAtomic: instance-unique tmp names (.tmp-pid-seq), a strict fsyncDir after rename so a successful persist is crash durable, and whitelist-based stale-tmp cleanup that never touches lock tmp files * fix(minidb): validate writes before any side effect, canonicalize values once - canonical value at the write boundary: the json codec re-parses the encoded bytes once and every downstream consumer (unique checks, secondary/compound/text indexes, dt extraction) sees exactly the persisted representation, so getter/toJSON/Proxy documents can no longer diverge between the index view and the storage view - reorder the set/batch pipeline so every fallible check happens before any visible side effect: prepare (key/ttl checks, encoding, canonical decode, index field extraction, tokenization) -> unique checks -> ensureMemoryFor eviction -> commit; a constraint failure now leaves the database untouched (no more evicted victims on rejected inserts), and applyOp is structurally pure against pre-validated data - tokenize at the prepare boundary: TextIndex gains prepareAdd/ addPrepared and the buildQueue carries validated key+tokens mutations instead of raw docs, so a throwing custom tokenizer can no longer poison the live view or the queue, and custom-tokenizer output is rejected per token over 0xffff bytes before it can permanently break postings rebuilds; prepared tokens are keyed by index instance so a same-name drop+create mid-write re-tokenizes instead of crossing tokenizers - strict batch structure validation: scanBatchOpRefs/decodeBatchOps reject unknown op types, out-of-bounds lengths, and trailing bytes (offset must equal body length), so a valid-CRC but malformed batch is skipped as a unit and counted via RecoveryInfo.corruptBatches instead of being partially applied Bench vs the stage-1 baseline: json write throughput regression is within the 5% budget (median ~2-4% depending on the measurement). * feat(minidb): add OpTracker drain primitive and atomic backup, harden tests - introduce the internal OpTracker (close gate + in-flight counter with enter/leave/close/whenIdle and reference-counted pause/resume) and drive every shutdown/drain path from it: WAL background syncs are tracked so close() waits out an in-flight sync before closing the fd, cluster lock-pool closeAll() closes the gates and drains busy callbacks before closing handles, and MiniDb writes pass a write gate - make backup() atomic with a defined linearization point: pause the write gate, drain in-flight writes (every acknowledged write is now included), copy to a sibling temp dir with per-file fsyncs, write the manifest last as the commit marker, and rename into place; failures clean up and leave no partial backup, and concurrent writes are rejected with BACKUP_IN_PROGRESS - reap emptied compound-index groups on remove (the groups map no longer grows monotonically), move the open-time mkdir behind the readOnly check so a read-only open of a missing directory fails with ENOENT instead of creating it, and never run a destructive rebuild for a read-only open failure (explicit or onLockFail fallback) - consolidate every review fault-injection repro into the formal suite behind deterministic barrier helpers (programmable writev/sync/ rename/tokenize hooks) and convert the six timing-based tests to barrier/tick-driven assertions; the .tmp repro scripts are removed The converted timing tests and the full suite pass 50 repeat runs (including under CPU load injection) with zero flakes. * feat(minidb): persist derived indexes as atomic generations, open from WAL delta - checkpoint the store, dt/secondary/compound indexes, and text dictionary/postings/docs into immutable generations under generations/g-NNNNNN published atomically (tmp build, per-file checksums and fsyncs, dir rename, CURRENT swap, strict dir fsyncs); the manifest records the format version, WAL/snapshot checkpoint anchors, per-index definition hashes, and codec/value-mode compatibility - open now loads the published generation and replays only the WAL delta after its checkpoint: no full value decode, corpus tokenization, or postings rewrite on a normal reopen (warm opens are 3.5-13.8x faster at 100k/1M records); a definition change rebuilds only the affected index, and corrupt generation files fall back to the previous generation or the legacy full recovery without ever touching the authoritative snapshot/WAL - build generations transactionally with compaction (rotation plus derived state publish as one unit, replacing the synchronous rebuildTextPostings tail), capture concurrent writes through a sealed op queue with byte/op caps, hard-link clean postings and the snapshot into the new generation, and repoint every live text base into the CURRENT generation after publish - cluster/read-only refresh watches CURRENT and the WAL watermark: pure generation publishes keep readers on incremental catch-up while rotations reopen onto the new generation; writers building the next generation never disturb readers of the current one - legacy databases open through the old path unchanged and gain their first generation in the background; OpenOptions.indexGenerations: false fully restores the pre-generation behavior * feat(minidb): workerize text-index builds and split MiniDb into facets - split the monolithic src/index.ts into facet modules (mini-db, types, value-codec, memory-guard, backup, query-engine, text-registry, wal-group, generation-builder/loader, write-path, read-path, index-admin, lifecycle, stats) and move text-index.ts to text-index/ - run corpus-scale text-index builds off the main thread via the bounded worker engine (src/worker/), exported through the new worker-runtime subpath, with inline fallback for small corpora and rollback switches - defer the open-time fallback text rebuild into a maintenance task; searches on a not-yet-committed base raise TextIndexBuildingError - add the unified maintenance scheduler, bounded async read surface, and a maintenance bench - kap-server search: switch to searchBoundedAsync and serve the building page while the index base rebuilds after fallback recovery - kimi-code: install the SEA-bundled minidb text-build worker at startup, bundle it via the native asset scripts, and add the startup-trace util plus the KIMI_TUI_INPUT_LATENCY debug probe * fix(minidb): treat win32 EPERM as unsupported directory fsync - extract isUnsupportedDirectoryFsyncError and cover win32 EPERM - drop the one-shot console.warn; stats.dirFsyncUnsupported carries the degraded state * fix(kap-server): harden search-index dispose and drain lifecycle - dispose() now closes an OpTracker gate and drains in-flight sync/refresh passes before closing the db, so no background write can hit a closed handle; the deleteSessionDocs loop and trailing stats write skip once the gate closes (review MoonshotAI#20) - drainGlobalSearchDisposals loops to a fixpoint so disposals registered while a drain is in flight are also awaited (review MoonshotAI#21) - pin the post-open failure semantics with a regression test: a failed text-index setup closes the handle and the next open reacquires the writer lock instead of self-locking read-only (review MoonshotAI#19) - export OpTracker from the minidb root for the search service's drain * chore: fix oxlint type-aware lint errors
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issue
No linked issue — this PR lands an internally planned, staged reliability & performance program for the embedded store behind the global message search; the problem is explained below.
Problem
The global message search (
POST /api/v1/search) is backed by a single embedded minidb database. Under sustained use, several compounding problems showed up:What changed
Fourteen commits, staged in reviewable steps (rebased onto current main; the read-model redesign absorbs main's newer consumers of the previous session-index contract):
1. Bounded hot paths and observability
2. Search read model and bounded query lifecycle
removewrite carried over from the previous contract).incompleteinstead of silently truncating; literal mode runs through a 2/3-gram candidate index with exact confirmation.3. Persistent index generations and workerized builds
generations/g-N+CURRENT); open loads the published generation plus a WAL delta instead of rebuilding the world, with full recovery as the automatic fallback.4. Write-failure and lock hardening
5. Lifecycle drain
Checklist
gen-changesetsskill, or this PR needs no changeset (no changesets — the changelog entries will be written at release time from the commit history).gen-docsskill, or this PR needs no doc update (no user-facing configuration or command changes).