Skip to content

feat(rpc): authoritative message terminals, metadata and ordered cold edits for the compact reducer (S03b-2b, part of #4272) - #4425

Open
code-yeongyu wants to merge 5 commits into
mainfrom
feat/4272-s03b2b-terminals
Open

code-yeongyu wants to merge 5 commits into
mainfrom
feat/4272-s03b2b-terminals

Conversation

@code-yeongyu

Copy link
Copy Markdown
Owner

Part of #4272. S03b-2b: authoritative message terminals, message metadata/usage/correlation, and ordered cold flows (edit, delete, compaction removal, branching) for the compact reducer (R3.C6.4), behind the OFF-by-default compact-updates flag. Stacked on #4414 (S03b-2a).

S03b-2b: measured size stop with a complete, green candidate

Part of #4272 (R3.C6.4). Worktree senpi-s03b2b, branch feat/4272-s03b2b-terminals.

DECISION (lane, 2026-10-27): option B approved

The lane approved the single PR as measured (493 added / 145 removed, net 348). The ~400 cap in the brief was the
lane's own; the PR rule is < ~600 production lines. Option A (split into 2b-1 / 2b-2) and option C (cut scope) are
not taken. Moved-lines note for reviewers: about 53 of the 493 added lines are publish / dropActive / block and
the active-state fields moved verbatim from compact-session-state.ts into the new compact-active-messages.ts,
because adding the message fold in place would have taken that file past the 250-line ceiling (HEAD 236; now 199).
Nothing was compressed to fit. The sections below are the measurement that decision was made on; the post-merge
state is recorded under "Merge of #4414's fold" at the end.

Result: stopped at the brief's size gate. The full S03b-2b scope is implemented and every gate is green, but it
measures 493 added / 145 removed formatted pure production LOC (net 348) against the ~400 cap. No commit, no
fragment, no push. HEAD is still 93698ed91c64cd24ccd0766a5dd28b3778481a72.

The candidate is left UNCOMMITTED in the worktree (16 paths) so a GO needs only a fragment and a commit. A safety
copy is archived: s03b2b-size-stop.patch, 65,346 bytes, SHA-256
3b6522592c1a163404af3b14d2a1f1b8021fec425ed821402feff56fc710c8a7, new files included;
git apply --check -R against the worktree exits 0.

Size (TypeScript 6.0.3 leaf-token line spans after oxfmt; comments and blank lines excluded)

Measured against HEAD with git diff -U0; new files count in full. Every file is at most 250.
Calibration: this method gives 236 for HEAD's compact-session-state.ts (2a's Babel count was 234).

File (src/modes/rpc/) Added Removed Whole file
compact-cold-operations.ts 110 38 152
compact-seeded-messages.ts (new) 112 0 112
compact-active-messages.ts (new) 97 0 97
compact-cold-flows.ts (new) 37 0 37
compact-session-state.ts 36 73 199
compact-transition.ts 24 7 77
compact-state-view.ts 19 9 62
compact-seeded-block.ts 17 3 175
compact-state-storage.ts 17 6 48
compact-seeded-block-view.ts 12 7 45
compact-seeded-draft.ts 6 1 55
compact-active-block.ts 4 0 58
compact-text-view.ts 2 1 42
Total 493 145

About 53 of the added lines are publish/dropActive/block and the active-state fields moved verbatim from
compact-session-state.ts (which loses 73 pure lines), because adding the message fold there would have taken that file
past 250 (HEAD 236). The cold queue's restructure accounts for 38 of the removed lines.

Measured split (each part independently green; 2b-2 stacks on 2b-1)

  • 2b-1, message terminals and metadata: 302 added. Transition types, the initial slot phase and frozenBlock,
    the draft's null removal, compact-seeded-messages.ts without installSettledMessage, compact-active-messages.ts,
    the session-state seeded_message path, message-field leaves in storage, and the views.
    Test: 4272-compact-terminal-authority.test.ts.
  • 2b-2, ordered cold edits/flows: 191 added. compact-cold-operations.ts (message_replace, message_install,
    runAll), compact-cold-flows.ts, installSettledMessage (29), the storage cold hook (11) and applyColdFlow (4).
    Test: 4272-cold-message-flows.test.ts.

Contract question (blocks the commit): A/B/C

  • A: split into 2b-1 (302) and 2b-2 (191) as measured above. Each is under ~400. This is mechanical from the
    green candidate, but costs a second review and gate round.
  • B (recommended): authorize the single PR at 493 added / net 348. It is complete and green at one head. The
    lane's PR rule is < ~600 production lines (BUDGET UPDATE, S03b2 brief). About 53 added lines are a pure move forced
    by the 250-per-file rule, and nothing was compressed.
  • C (not recommended): squeeze under 400 by dropping message_install (branch) or the one-slot runAll. That
    narrows the approved scope: a compaction of more than 128 messages would then hit capacity partway.

RED -> GREEN (contract test ids)

The RED run was on unchanged production, exit 1, 8/8 failed (red.log). GREEN: the same tests plus two pinned
additions, 65/65 with their neighbours (green-1.log), then inside the full serial run.

Test RED (unchanged production) GREEN
T6 owns message metadata, usage and correlation from the seeded start through indexed updates behavior: view empty / no message fields pass
T6 a terminal rebuilds rewritten blocks and drops every index at or above rewrittenFrom behavior: 3 stale blocks kept pass
T6 message_end repairs an invalidated block from the terminal without its abandoned stream behavior: stale raw and preview kept pass
T6 installs direct error and toolResult terminals with no start and binds their first ordinal behavior: expected [] to match pass
T6 rejects a restart after streaming, malformed cutoffs and a foreign epoch atomically behavior: no typed rejection pass
T6 edits a settled message after acceptance, keeping its persistence and every held cut behavior: edit deleted the row (old run() treated it as a remove) pass
T6 maps user delete, compaction removal and branch replacement to ordered typed outcomes applyColdFlow is not a function (new entry point; its per-op semantics RED in the edit test) pass
T6 runs a large flow as one ordered unit holding a single capacity slot same absence pass

Mutation probe (mutation.log): removing the ended guard on raw reuse and the correlation carry on edit fails exactly
the two pinned assertions: the nested done with no cutoff in the repair test, and the seeded-row correlation in the
edit test. The source was restored byte-identical afterwards.

Gates on the candidate (nice 10, CI=1, TMPDIR=delta-senpi/tmp, five session variables removed, own caches)

Each gate waited until the 5-minute load was below 60 (s03b2b-run.mjs polls every 60 s).
Logs: senpi-s03b2b/local-ignore/qa-evidence/20261027-s03b2b/.

Gate Exit
bun run build 0
Required serial suites (4272-, 4349-, rpc-compact-content-*, no-sync-in-session-path) after build 0: 61 files / 270 tests, 222 s
node_modules/.bin/tsc --noEmit 0
node scripts/lint-ratchet.mjs 0, every count matches baseline
oxlint --deny-warnings on changed files 0
oxfmt --check on changed files 0
root bun run check 0
git diff --check 0

An earlier serial run before the build (suites.log) had 13 failures in worker/epoch/bootstrap suites. All came from
the unbuilt @earendil-works/pi-ai dist (provider-scope.js missing, so workers quarantined), not from this change.
The built rerun is the evidence. Flag off is byte-identical: no production path constructs this owner yet, and
json-event.ts and structured-session-records.ts are unchanged. No new sync call was added; the no-sync suite passes.

What the candidate implements

  • seeded_message transition: outer message_start/message_end and nested start/done/error.
    • A start installs public metadata and its content as initial slots, which that index's own block start takes
      over. A start after streamed blocks is rejected invalid_phase.
    • done/error/message_end are authoritative: the whole message is rebuilt on a fresh block root, so no index the
      terminal no longer names survives.
    • Raw tool spelling is kept only from an ENDED block below rewrittenFrom that is not in invalidatedBlocks, so an
      abandoned stream's raw text and preview never outlive the terminal.
    • The outer message_end settles as pending in the same step. Direct error and toolResult terminals need no start.
    • A message-creating record validates and commits R1 identity, so a reused first ordinal is rejected
      duplicate_or_stale_start. It also calls cold.activate (see below). A foreign epoch is rejected wrong_epoch;
      malformed cutoffs are rejected invalid_terminal.
  • seeded_block gains optional usage and messageMetadata (null removes a field), folded in the same step.
  • Correlation binds once when the message is created and is carried into the settled row and through edits.
  • Cold operations, ordered after acceptance (RULING-33 condition 2):
    • message_replace (edit): the row must exist; persistence and correlation are kept.
    • message_install (branch): the row is installed whole with the given persistence, present or not.
    • runAll runs a whole flow as ONE FIFO unit holding one capacity slot, with one typed outcome per operation.
    • applyColdFlow maps the flows: edit -> replace; user_delete -> remove; compaction -> ordered removes;
      branch -> removes, then installs.
    • A live target answers active_message and remains synchronous acceptance work (a live user delete is the accepted
      message_remove transition).

Coordination with the #4414 fold (heads-up received mid-run)

  • The shared activation helper is not on this base. acceptMessage's creation path calls this.cold.activate
    directly, like the legacy start(). The fold's helper must cover this path (compact-session-state.ts, acceptMessage).
  • The initial phase is a message-start slot, not an open block. The fold's rule that a native re-insert on an open
    block is duplicate_block should exempt initial slots, or a start that seeded content followed by a native insert
    at that index will be refused. Untouched here: the 2a seeded_block start path and insert semantics.

LOW disposition

Item Disposition
13 Carried: worker readiness deadline support not touched.
15 Carried: value page packing and string budget wording are substrate (S03b-4).
16 Carried to the #4398 fold (test-only precondition, overlay scan).
17-rest Carried: tiny-delta structural constant factor is substrate.
18 Carried to S05/S03b-3 (post-repair file identity).
19a (producer ?? comment) Carried: it is in S02c's packages/ai file, not the reducer.
19b (gap/repeated-start invalidation needs terminal repair) Folded: message_end with invalidatedBlocks, and a nested done with no cutoff, restore the block from the terminal without its abandoned stream (the repair test, mutation-checked).

What S03b-3 must wire

  1. Lane record mapping:
    • indexed message_update -> seeded_block {event, usage, messageMetadata};
    • the three controls -> seeded_block {event};
    • message_start/message_end and nested start/done/error -> seeded_message, with the correlation at creation, the
      gate's invalidatedBlocks, and the capture's rewrittenFrom on terminals.
  2. Real producers of the four flows -> applyColdFlow after their accepted transcript transition, with the lane cut:
    edit, user delete, the compaction removal list, and branch removal + install.
  3. From 2a: one acceptance unit; in-tree producer conformance (nonce, outputOrdinal, firstOrdinal, one numbering);
    current-epoch-only validation against the S05 registry; supported construction capture with zero peers and
    selection OFF; shared frozen component roots for S06/S07.

Assumptions and open items

  • CompactValues rejects a nested undefined property (unsupported_value). Real public messages can carry
    undefined optional fields (for example toolResult details). S03b-3 must normalize them, or S03b-4 should skip
    undefined object properties as JSON does. The fixtures here do not hit it.
  • A later record's correlation never rewrites the bound one (immutable at creation).
  • A terminal copies its content once per terminal (done, then message_end) and reclaims the old root asynchronously;
    X07 accounting for that is S03b-3/S12 evidence, not claimed here.

Merge of #4414's fold (origin/feat/4272-s03b2-reducer @ 851f442)

  • S03b-2b commit: f4a67ca750 (author and committer YeonGyu-Kim code.yeon.gyu@gmail.com), fragment changes/unreleased/s03b2b.md.
  • Merged with --no-ff. Conflicts in compact-seeded-block.ts and compact-session-state.ts only.
  • compact-seeded-block.ts: kept the fold's seedOffset (typed invalid_seed_frontier), the native-end kind check and the
    native-insert duplicate_block, plus this branch's exported seedBlock. The insert rule now EXEMPTS a start-content
    placeholder slot (phase === "initial"), exactly as seeded starts already did; the session-state starting test treats
    an insert over a placeholder as a start, so the first-ordinal guard still runs and commits.
  • compact-session-state.ts: kept this branch's structure (publish/dropActive/block live in compact-active-messages.ts) and
    dropped both direct this.cold.activate calls (legacy start() and acceptMessage). The fold's rule "activate on first
    entry into active" now lives in CompactActiveMessages.publishRow, the single writer of active, which both
    publish() and install() (the seeded_message path) go through. git grep cold.activate finds exactly one call site.

New tests (RED on the mechanical resolution, GREEN after the fix)

Test RED GREEN
4272-cold-message-flows: seeded_message incarnation (start_then_end) voids queued cold work stale status applied, expected active_message pass
4272-cold-message-flows: seeded_message incarnation (end_only) voids queued cold work stale status applied, expected active_message pass
4272-seeded-block-controls: native insert takes over a start-content slot; real duplicates (ended, open, native) rejected atomically duplicate_block: 0 on the placeholder pass

Guard-removal check: removing the native-insert duplicate_block guard fails exactly the new controls test
("expected function to throw"); restored byte-identical.

Gates after the merge (staged tree; nice -n 10, load < 60, logs m-*.log)

build 0; serial suites 0 (61 files / 286 tests); tsc 0; lint-ratchet 0 (every count matches baseline); oxlint
--deny-warnings 0 and oxfmt --check 0 on every file changed since 93698ed; root bun run check 0; git diff --cached --check 0.

LOC after the merge, versus the new base 851f442

Added 497 / removed 148 / net 349. Every file at most 250.

File (src/modes/rpc/) Added Removed Whole
compact-seeded-messages.ts 112 0 112
compact-cold-operations.ts 110 38 152
compact-active-messages.ts 98 0 98
compact-cold-flows.ts 37 0 37
compact-session-state.ts 37 75 198
compact-transition.ts 24 7 78
compact-seeded-block.ts 19 4 192
compact-state-view.ts 19 9 62
compact-state-storage.ts 17 6 48
compact-seeded-block-view.ts 12 7 45
compact-seeded-draft.ts 6 1 55
compact-active-block.ts 4 0 58
compact-text-view.ts 2 1 42

… edits for the compact reducer (S03b-2b, part of #4272)
…er-exempt native insert (part of #4272)

Merge origin/feat/4272-s03b2-reducer @ 851f442. Activation on first entry into the active set moves
to CompactActiveMessages.publishRow, the single writer of active, so seeded_message incarnations are
activated too; the direct cold.activate calls are removed. A native insert over a start-content
placeholder slot is a takeover, not a duplicate; any other existing block stays a typed duplicate_block.
@code-yeongyu code-yeongyu added the no-changelog No changelog required label Oct 27, 2026
@code-yeongyu
code-yeongyu marked this pull request as ready for review October 27, 2026 05:32
MoerAI pushed a commit to MoerAI/senpi that referenced this pull request Oct 27, 2026
…he stack merge (part of code-yeongyu#4272)

The merge of code-yeongyu#4374 onto code-yeongyu#4425 compiled S05's CompactSettledMessages against the S03a index API: synchronous has() and a generic FrozenIndex no longer exist, so every append threw. Keep the exact membership S05 relied on in a resident key set (its pruning stays carried item 9), and let a retired lane freeze without asking its retired paged store for a cut.

(cherry picked from commit e0fea84)
Base automatically changed from feat/4272-s03b2-reducer to main October 27, 2026 18:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog No changelog required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant