Skip to content

feat(rpc): persistent bounded index pages and compaction for the compact consumer (S03b-1b-I-b, part of #4272) - #4386

Merged
code-yeongyu merged 6 commits into
mainfrom
feat/4272-s03b1b-ib-index
Oct 27, 2026
Merged

code-yeongyu merged 6 commits into
mainfrom
feat/4272-s03b1b-ib-index

Conversation

@code-yeongyu

Copy link
Copy Markdown
Owner

Part of #4272. S03b-1b-I-b: persistent bounded index pages, refcounted AVL, bounded compaction and the real settled-index integration (R3.C6.4), behind the OFF-by-default compact-updates flag. Stacked on #4385 (S03b-1b-I-a); wires its codec and removes its staging guard.

Rulings: RULING-26/26a (condition 4: oversized-row RED, bounded memory, peak-resident-pages), RULING-34 (+ bugwatch split order and size acceptance). Exact resident membership retained at this head (RULING-34 option a); live-set membership + high-water + cold ops are S03b-1b-II (RULING-33).

S03b-1b-I-b: persistent index pages and settled consumer integration

Part of #4272. Local commit:
7161c1310b49984ebe5ec8b9b0ca6b2cbf88b599.
Parent: 889849b243af9b4e0fc5f950c9151b9c92364e25 (S03b-1b-I-a).
No push. Author and committer: YeonGyu-Kim code.yeon.gyu@gmail.com.

Scope and ordering

This completes the approved codec-first 1b-I split. The real
CompactSessionState settled index uses the codec through CompactContentIndex;
it is not a test-only index. This commit deletes the staging guard from 1b-I-a
as part of that wiring.

RULING-34 option (a) is used: exact resident membership is retained for cold
settled keys under today's caps. Duplicate starts and unknown status/removal
operations still fail without sequence or storage changes at both commit heads.
A valid update of a cold settled message remains accepted. There is no live-set,
Bloom-filter, or high-water substitute in this increment. Rollout stays OFF.

Persistent counted AVL roots share base and overlay paths across cuts. Index
pages contain bounded key/value references, not complete oversized keys or
structured rows. The compactor selects at most 128 overrides, reads at most the
touched page and one small page, and publishes synchronously after its I/O.
Its input page leases survive through the operation; output buffers participate
in peak-page accounting. Failed merges and stale rows release only the owners
they actually acquired. Direct page operations retain a cut until they unwind
and reject released or retired results, including the in-memory completion path.

Review focus: the counted AVL (refcounts and rebalancing under shared roots)
and the bounded compactor (held cuts, stale publication, failed merges).

RED and GREEN by deferred-table row

Anchors refer to the S03a input revision in s03b1a-pr-notes.md.

Original deferred row Regression and evidence Closure at this head
compact-content-index.ts:34-41,87-105 4272-persistent-index-pages.test.ts: predecessor freeze visited 2,048 ownership entries against a <=32 bound (RED, exit 1). GREEN covers shared-root freezing, full bounded traversal, a held disk-page read across release, and the retirement completion boundary. Freeze copies and pin walks removed; direct bounded page API and operation leases supplied. Exact resident key membership remains by RULING-34. entries() is still explicitly whole-view compatibility assembly.
compact-content-index.ts:118-172 4272-bounded-index-compaction.test.ts: the predecessor let producer mutation change a frozen nested row (RED, exit 1). GREEN proves the frozen value and newer patch survive. A separate deterministic late-merge-failure test proves skipped stale rows retain no obsolete payload owner and partial merges reclaim temporary leaves/nodes; that second case is preservation coverage, not an absent-API RED claim. Selection/materialization is bounded to the described batch and page working set; no all-row compaction or all-page output array.
compact-content-index.ts:182-211 4272-persistent-index-pages.test.ts: a one-key publication over 1,024 base rows stays within 4,096 new path nodes and preserves both old and new frozen values. Publication changes persistent paths instead of cloning the complete base map/list.
compact-content-index.ts:213-218 4272-bounded-index-pages.test.ts plus 4272-bounded-index-memory.test.ts: encoded byte/count ceilings, decoded page leases, and large traversal peak-page assertions pass. Reads use <=128 descriptors and <=65,536 encoded bytes per page, with no complete key-set filtering copy.
compact-index-pages.ts:14-29 4272-bounded-index-pages.test.ts: predecessor produced a 224,045-byte page against the 65,536 ceiling (RED, exit 1). GREEN preserves the oversized row/key through references. A raw oversized row is rejected with CompactWriteError(size_limit) before yield; exact 65,536-byte and 128/129-entry boundaries pass. Oversized row acceptance and eager all-pages packing removed; no truncation.
compact-session-state.ts:49-65,81-82,110-132 4272-bounded-current-components.test.ts preserves 1,024 settled block descriptors and frozen persistence status through a cold update in the real consumer. Nested ownership and oversized values are covered by the compaction/page tests above. Structured settled values now spill and remain immutable. Active collection freeze/copy migration remains 1b-II, as explicitly split by RULING-34.
compact-session-state.ts:136-155,253-277 Existing compatibility and exact-membership regressions pass. This increment does not claim the one-block active-update/collection-copy proof. 1b-II owns active header walks and seal-time block sorting/materialization. Whole legacy view assembly remains a separately charged compatibility operation.

The additional RAM-retirement RED returned a successful page (undefined from
the test's rejection capture) after retirement had begun. The owner-validity
check now makes it fail with CompactReadError(retired). The final targeted
publication/ownership run passed 4 files / 9 tests, exit 0.

Bounded memory and retained costs

4272-bounded-index-memory.test.ts covers 1,024 rows. After spill, structural
payload residency is zero; resident index nodes stay within the asserted
2 * 1024 bound. Traversal visits all rows with <=128
descriptors and <=64 KiB per page and peak resident index pages <=2. After cut
release and removal of all current rows, structural counters and reclaim debt
return to zero. The separate real-consumer fixture verifies all 1,024 blocks
and the frozen persistence status across a cold status update.

These are logical ownership/page-working-set assertions, not RSS measurements.
Resident exact membership, AVL headers, page key lists, and text/ownership
navigation headers remain counted. The compatibility entries()/whole snapshot
path still assembles the requested whole result. No constant-memory full legacy
view or constant-RSS claim is made.

Formatted production size

500 added pure production LOC, relative to 1b-I-a. Together the two commits
add 805 pure production LOC (305 + 500). This supersedes the earlier one-PR
size waiver; the approved reversed split keeps both increments below ~550.
The final ownership and page-buffer lifetime corrections are included in these
formatted counts.

RPC source file Added pure LOC Whole-file pure LOC
compact-content-index.ts 193 234
compact-index-tree.ts 128 128
compact-index-compactor.ts 161 161
compact-index-pages.ts 13 17
compact-read-lease.ts 4 38
compact-session-state.ts 1 248

Other changed source/test files, under packages/coding-agent/test/suite/:

File Whole-file pure LOC
compact-storage-support.ts 56
regressions/4272-bounded-current-components.test.ts 44
regressions/4272-bounded-index-compaction.test.ts 75
regressions/4272-bounded-index-memory.test.ts 42
regressions/4272-bounded-index-pages.test.ts 39
regressions/4272-persistent-index-pages.test.ts 128
regressions/4272-text-operation-leases.test.ts 100
rpc-compact-content-index.test.ts 154
rpc-compact-content-store-progress.test.ts 195
rpc-compact-content-store.test.ts 250
regressions/4272-structured-value-codec-test-only.test.ts Deleted (20 at A)

Every changed source/test file is <=250 pure LOC. Test adaptations target
representation accounting and exact I/O barriers; retained content assertions
were not weakened.

Contract closure and next increment

  • RULING-26 Q3: the bounded structured value/index storage APIs are supplied;
    the remaining collection migration is not claimed complete.
  • RULING-26a condition 1: sequential, independently testable OFF-gated commits.
    Conditions 2 and 3 retain 1a's shared ownership/lease coverage; the residual
    collection costs are listed above. Condition 4's oversized-row RED,
    bounded-memory test, and large-collection peak-page assertion ship here.
  • RULING-34 ordering condition: exact resident cold membership at both heads,
    with consumer-level duplicate/unknown-op rejection coverage.
  • RULING-33 live-set membership, epoch high-water duplicate_or_stale_start,
    and cold-operation enumeration/non-acceptance are 1b-II, alongside the
    remaining active/settled component-root migration.
  • RULING-26 Q1/Q2/Q4, the full reducer and lane integration, S07b old-display
    consumer/replay removal, and S12 worker transfer remain their later steps.
  • The feat(rpc): pending/failed terminals and message-correlated append joins (S05, part of #4272) #4374 S05 index-row pruning LOW remains S03b-2/3; its module is not on this
    branch.

Unchanged #4378 notes: compact-text-view.ts:32-35 reads compatibility chunks
sequentially; compact-spool-storage.ts retirement waits for in-flight reads
without a deadline by design; compact-session-state.ts resident text-index
bytes remain report-only. The two requested code LOWs landed in 1b-I-a.

No unresolved contract question remains within the approved 1b-I scope.

Exact-head verification

The build, test, full-check, CLI, and parity commands ran after commit
7161c1310b49984ebe5ec8b9b0ca6b2cbf88b599, with the brief's sanitized child
environment, load gate, and nice -n 10.

Gate Exit / result
bun run build 0; all workspace phases and the coding-agent bundle.
Focused 4272-, compact-store, and structured-sink suites 0; 19 files / 65 tests.
Standalone no-sync-in-session-path.test.ts 0; 1 file / 6 tests; baseline unchanged.
bun run check 0; oxlint, oxfmt, ratchet, dependencies/imports/entry graphs/locks, both TypeScript projects, and browser smoke.
Separate changed-file oxlint and oxfmt checks 0 each; all 25 changed source/test files.
Full-stack git diff --check 0.
Real RPC CLI rpc-drive.mjs --self-test 0; 4/4, offline sandbox, real authentication unchanged.
Predecessor and new consumer parity drivers 0 each; identical five frozen views, 131,809 UTF-8 bytes, sequence 7, cleanup 0.
Changelog gate against A 1; only s03b1b-Ib.md needs its assigned PR number.
Changelog gate against origin/feat/4272-s03b1-storage-substrate 1; only the two required temporary fragment filenames need their assigned PR numbers.

Parity SHA-256:
a92cccae1ef3a4c959ab68d75a87ae80d77298f5ce39c6ca1e47ae963d29f701.
The schedule covers append, replacement, split/lone surrogates, pending/failed
persistence, removal, and retained frozen views. The focused suites also retain
the unchanged flag-OFF byte-identity writer fixture. This is not an S07b
old-display-consumer parity claim.

The configured Husky launcher was absent; no hook was bypassed. The full gates
were run explicitly. Final lease/index LSP diagnostics were clear; earlier
individual test-file diagnostic requests had timed out. No external review or
merge is claimed.

Final tracked working-tree status is clean. Both commits are local. The A-only
verification worktree was removed and its absence verified.

@code-yeongyu code-yeongyu added the no-changelog No changelog required label Oct 26, 2026
@code-yeongyu
code-yeongyu marked this pull request as ready for review October 26, 2026 13:48
Base automatically changed from feat/4272-s03b1b-ia-codec to main October 27, 2026 07:04
@code-yeongyu code-yeongyu added no-changelog No changelog required and removed no-changelog No changelog required labels Oct 27, 2026
@code-yeongyu
code-yeongyu merged commit fe6ab06 into main Oct 27, 2026
51 of 52 checks passed
@code-yeongyu
code-yeongyu deleted the feat/4272-s03b1b-ib-index branch October 27, 2026 08:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog No changelog required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant