Repository navigation
feat(rpc): persistent bounded index pages and compaction for the compact consumer (S03b-1b-I-b, part of #4272) - #4386
Merged
Merged
Conversation
Co-authored-by: sisyphus-dev-ai <sisyphus-dev-ai@users.noreply.github.com>
…feat/4272-s03b1b-ib-index
code-yeongyu
marked this pull request as ready for review
October 26, 2026 13:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #4272. S03b-1b-I-b: persistent bounded index pages, refcounted AVL, bounded compaction and the real settled-index integration (R3.C6.4), behind the OFF-by-default compact-updates flag. Stacked on #4385 (S03b-1b-I-a); wires its codec and removes its staging guard.
Rulings: RULING-26/26a (condition 4: oversized-row RED, bounded memory, peak-resident-pages), RULING-34 (+ bugwatch split order and size acceptance). Exact resident membership retained at this head (RULING-34 option a); live-set membership + high-water + cold ops are S03b-1b-II (RULING-33).
S03b-1b-I-b: persistent index pages and settled consumer integration
Part of #4272. Local commit:
7161c1310b49984ebe5ec8b9b0ca6b2cbf88b599.Parent:
889849b243af9b4e0fc5f950c9151b9c92364e25(S03b-1b-I-a).No push. Author and committer: YeonGyu-Kim code.yeon.gyu@gmail.com.
Scope and ordering
This completes the approved codec-first 1b-I split. The real
CompactSessionStatesettled index uses the codec throughCompactContentIndex;it is not a test-only index. This commit deletes the staging guard from 1b-I-a
as part of that wiring.
RULING-34 option (a) is used: exact resident membership is retained for cold
settled keys under today's caps. Duplicate starts and unknown status/removal
operations still fail without sequence or storage changes at both commit heads.
A valid update of a cold settled message remains accepted. There is no live-set,
Bloom-filter, or high-water substitute in this increment. Rollout stays OFF.
Persistent counted AVL roots share base and overlay paths across cuts. Index
pages contain bounded key/value references, not complete oversized keys or
structured rows. The compactor selects at most 128 overrides, reads at most the
touched page and one small page, and publishes synchronously after its I/O.
Its input page leases survive through the operation; output buffers participate
in peak-page accounting. Failed merges and stale rows release only the owners
they actually acquired. Direct page operations retain a cut until they unwind
and reject released or retired results, including the in-memory completion path.
Review focus: the counted AVL (refcounts and rebalancing under shared roots)
and the bounded compactor (held cuts, stale publication, failed merges).
RED and GREEN by deferred-table row
Anchors refer to the S03a input revision in
s03b1a-pr-notes.md.compact-content-index.ts:34-41,87-1054272-persistent-index-pages.test.ts: predecessor freeze visited 2,048 ownership entries against a <=32 bound (RED, exit 1). GREEN covers shared-root freezing, full bounded traversal, a held disk-page read across release, and the retirement completion boundary.entries()is still explicitly whole-view compatibility assembly.compact-content-index.ts:118-1724272-bounded-index-compaction.test.ts: the predecessor let producer mutation change a frozen nested row (RED, exit 1). GREEN proves the frozen value and newer patch survive. A separate deterministic late-merge-failure test proves skipped stale rows retain no obsolete payload owner and partial merges reclaim temporary leaves/nodes; that second case is preservation coverage, not an absent-API RED claim.compact-content-index.ts:182-2114272-persistent-index-pages.test.ts: a one-key publication over 1,024 base rows stays within 4,096 new path nodes and preserves both old and new frozen values.compact-content-index.ts:213-2184272-bounded-index-pages.test.tsplus4272-bounded-index-memory.test.ts: encoded byte/count ceilings, decoded page leases, and large traversal peak-page assertions pass.compact-index-pages.ts:14-294272-bounded-index-pages.test.ts: predecessor produced a 224,045-byte page against the 65,536 ceiling (RED, exit 1). GREEN preserves the oversized row/key through references. A raw oversized row is rejected withCompactWriteError(size_limit)before yield; exact 65,536-byte and 128/129-entry boundaries pass.compact-session-state.ts:49-65,81-82,110-1324272-bounded-current-components.test.tspreserves 1,024 settled block descriptors and frozen persistence status through a cold update in the real consumer. Nested ownership and oversized values are covered by the compaction/page tests above.compact-session-state.ts:136-155,253-277The additional RAM-retirement RED returned a successful page (
undefinedfromthe test's rejection capture) after retirement had begun. The owner-validity
check now makes it fail with
CompactReadError(retired). The final targetedpublication/ownership run passed 4 files / 9 tests, exit 0.
Bounded memory and retained costs
4272-bounded-index-memory.test.tscovers 1,024 rows. After spill, structuralpayload residency is zero; resident index nodes stay within the asserted
2 * 1024bound. Traversal visits all rows with <=128descriptors and <=64 KiB per page and peak resident index pages <=2. After cut
release and removal of all current rows, structural counters and reclaim debt
return to zero. The separate real-consumer fixture verifies all 1,024 blocks
and the frozen persistence status across a cold status update.
These are logical ownership/page-working-set assertions, not RSS measurements.
Resident exact membership, AVL headers, page key lists, and text/ownership
navigation headers remain counted. The compatibility
entries()/whole snapshotpath still assembles the requested whole result. No constant-memory full legacy
view or constant-RSS claim is made.
Formatted production size
500 added pure production LOC, relative to 1b-I-a. Together the two commits
add 805 pure production LOC (305 + 500). This supersedes the earlier one-PR
size waiver; the approved reversed split keeps both increments below ~550.
The final ownership and page-buffer lifetime corrections are included in these
formatted counts.
compact-content-index.tscompact-index-tree.tscompact-index-compactor.tscompact-index-pages.tscompact-read-lease.tscompact-session-state.tsOther changed source/test files, under
packages/coding-agent/test/suite/:compact-storage-support.tsregressions/4272-bounded-current-components.test.tsregressions/4272-bounded-index-compaction.test.tsregressions/4272-bounded-index-memory.test.tsregressions/4272-bounded-index-pages.test.tsregressions/4272-persistent-index-pages.test.tsregressions/4272-text-operation-leases.test.tsrpc-compact-content-index.test.tsrpc-compact-content-store-progress.test.tsrpc-compact-content-store.test.tsregressions/4272-structured-value-codec-test-only.test.tsEvery changed source/test file is <=250 pure LOC. Test adaptations target
representation accounting and exact I/O barriers; retained content assertions
were not weakened.
Contract closure and next increment
the remaining collection migration is not claimed complete.
Conditions 2 and 3 retain 1a's shared ownership/lease coverage; the residual
collection costs are listed above. Condition 4's oversized-row RED,
bounded-memory test, and large-collection peak-page assertion ship here.
with consumer-level duplicate/unknown-op rejection coverage.
duplicate_or_stale_start,and cold-operation enumeration/non-acceptance are 1b-II, alongside the
remaining active/settled component-root migration.
consumer/replay removal, and S12 worker transfer remain their later steps.
branch.
Unchanged #4378 notes:
compact-text-view.ts:32-35reads compatibility chunkssequentially;
compact-spool-storage.tsretirement waits for in-flight readswithout a deadline by design;
compact-session-state.tsresident text-indexbytes remain report-only. The two requested code LOWs landed in 1b-I-a.
No unresolved contract question remains within the approved 1b-I scope.
Exact-head verification
The build, test, full-check, CLI, and parity commands ran after commit
7161c1310b49984ebe5ec8b9b0ca6b2cbf88b599, with the brief's sanitized childenvironment, load gate, and
nice -n 10.bun run build4272-, compact-store, and structured-sink suitesno-sync-in-session-path.test.tsbun run checkgit diff --checkrpc-drive.mjs --self-tests03b1b-Ib.mdneeds its assigned PR number.origin/feat/4272-s03b1-storage-substrateParity SHA-256:
a92cccae1ef3a4c959ab68d75a87ae80d77298f5ce39c6ca1e47ae963d29f701.The schedule covers append, replacement, split/lone surrogates, pending/failed
persistence, removal, and retained frozen views. The focused suites also retain
the unchanged flag-OFF byte-identity writer fixture. This is not an S07b
old-display-consumer parity claim.
The configured Husky launcher was absent; no hook was bypassed. The full gates
were run explicitly. Final lease/index LSP diagnostics were clear; earlier
individual test-file diagnostic requests had timed out. No external review or
merge is claimed.
Final tracked working-tree status is clean. Both commits are local. The A-only
verification worktree was removed and its absence verified.