[stacked on #2148 ← #2153 ← #2136 ← #2124] Pin provider baselines: corpus row snapshots, timeline perf, permission matrix - #2121
Merged
SawyerHood merged 8 commits intoAug 21, 2026
Conversation
|
🚨 SLOP COP 🚨 · I am SlopCop. I am reviewing this pull request under the review rule. I will check security, quality, performance, architecture, duplication, and end-to-end behavior. |
![bb-slop-cop[bot]](https://avatars.githubusercontent.com/in/4577281?s=60&v=4){kind=small}
| `full walk p50 ${round(p50)} ms (samples ${durations.map((value) => round(value)).join(", ")})\n`, | ||
| ); | ||
| expect(last.length).toBeGreaterThan(1); | ||
| expect(p50).toBeLessThan(SYNTHETIC_CEILING_MS); |
There was a problem hiding this comment.
🚨 slopcop/review — The fixed 750 ms limit makes the normal suite unstable.
I ran this test three times on the unchanged head. One run failed with a p50 of 823.9 ms. The other p50 values were 157.2 ms and 206.9 ms. Please move this check to a controlled benchmark runner, or use a limit and retry policy based on the slowest supported runner.
Collaborator
Author
There was a problem hiding this comment.
Fixed. The benchmark now gates on the minimum of the five walks instead of the p50, and the ceiling is 1,500 ms (~10× the local minimum of 150–170 ms). A loaded runner only trips it when every sample is slow; a quadratic or unbounded regression still lands in whole seconds.
| // change in machine load lands on the calibration and the thread alike. | ||
| const samples = sample( | ||
| (): InterleavedSample => ({ | ||
| calibrationMs: sumProfileDurations(walkSynthetic(synthetic)), |
There was a problem hiding this comment.
🚨 slopcop/review — The calibration runs the same timeline path as the code under test.
A uniform timeline regression raises both values. Their ratio can stay stable, so the 10% gate can pass. Please calibrate with unrelated work. Otherwise, gate raw times on a controlled runner and keep this ratio as diagnostic data.
Collaborator
Author
There was a problem hiding this comment.
Agreed, that was a real blind spot. The calibration is now a fixed CPU workload with no timeline code in it (json-sort-v1: JSON parse/stringify over a deterministic 1.6 MB document, a 150k-element sort, and a string split), so a uniform regression in the build path moves the ratio. Raw p50/p95 stay in the baseline and the table as diagnostics.
| }), | ||
| }), | ||
| ); | ||
| const calibrationMinMs = Math.min( |
There was a problem hiding this comment.
🚨 slopcop/review — The ratio does not pair interleaved samples.
This code divides the lowest corpus result by the lowest calibration result. Different attempts can produce those values. This does not cancel per-sample load as the comment claims. Please calculate a ratio for each sample, then aggregate those paired ratios.
Collaborator
Author
There was a problem hiding this comment.
Fixed. Each sample now runs calibration, latest page, and full walk back to back, the ratio is computed per sample, and normalizedMin is the minimum of those paired ratios (normalizedP50 is recorded too). The comment was overstating what the old code did.
Collaborator
Author
There was a problem hiding this comment.
Correction to the above after measuring: the gate now uses min(build) ÷ min(calibration) over the interleaved samples, not the minimum of the per-sample ratios. Per-sample ratio minima turned out to be biased low — any stall on the calibration side of one pair produces a spuriously small ratio and the minimum picks exactly that pair, which minted baselines later runs could not match (one thread read 1.7× on a clean re-run). Each side's minimum discards its own contended samples; interleaving keeps both minima inside the same short window. The per-sample ratio minimum is still recorded as pairedRatioMin for diagnostics. With this estimator, compare on the same commit passed 20/20 threads on the first attempt (0.80–1.09× of baseline) at load 6–9 on 16 cores.
| const corpusDir = resolveProviderCorpusDir() ?? ""; | ||
| const snapshotsDir = path.join(corpusDir, "snapshots"); | ||
| const baselinePath = path.join(snapshotsDir, "perf-baseline.json"); | ||
| const baseline: PerfBaseline | null = |
There was a problem hiding this comment.
🚨 slopcop/review — Compare mode accepts incompatible baseline settings.
The file stores samplesPerThread and calibrationEventCount, but compare mode never checks them. A later constant change can compare different measurements. Please compare both fields with the current constants and require a baseline update after a mismatch.
Collaborator
Author
There was a problem hiding this comment.
Fixed. The baseline carries a gate block (samplesPerThread, calibration kind) and compare mode refuses a baseline whose settings differ from the suite's constants with a message that says to rewrite it.
| provider: string, | ||
| threadId: string, | ||
| ): string { | ||
| return path.join(snapshotsDir, "rows", provider, `${threadId}.json`); |
There was a problem hiding this comment.
🚨 slopcop/review — An unvalidated provider can move snapshot access outside the corpus directory.
The manifest accepts any nonempty provider string. A value with ../ segments can move this read or write outside snapshots/rows. Please require one safe path segment. Also reject a resolved path outside the rows root.
Collaborator
Author
There was a problem hiding this comment.
Fixed at the boundary: the reader's zod schema now requires thread and provider ids to be one safe path segment (^[A-Za-z0-9][A-Za-z0-9_-]*$) in both the manifest and meta.json, and snapshotFilePath rejects a resolved path outside snapshots/rows as a second guard.
| throw new Error(`Corpus thread ${threadId} not found under ${threadsDir}`); | ||
| } | ||
|
|
||
| export function loadCorpusThread(threadId: string): CorpusThread { |
There was a problem hiding this comment.
🚨 slopcop/review — The loader does not link the manifest, metadata, and event rows.
It searches all providers by ID. It does not verify the metadata ID, provider, manifest count, or each event thread ID. A stale corpus can create a valid but incorrect baseline. Please load through the manifest entry and validate all cross-file identities.
Collaborator
Author
There was a problem hiding this comment.
Fixed. loadCorpusThread now resolves the directory through the manifest entry and fails unless meta.json names the same thread id and provider, the reasons agree, every event row's thread_id is the thread, and the row count matches both the manifest and meta.json. The full corpus still loads (307 threads, 330,626 rows).
| maxInlineOutputChars: DEFAULT_MAX_INLINE_OUTPUT_CHARS, | ||
| maxSeq, | ||
| page: args.page, | ||
| providerDisplayName: |
There was a problem hiding this comment.
🚨 slopcop/review — The route-equivalent harness copies production policy.
The route gets provider labels, plan commands, flags, and limits from the registry and configuration. These copies can drift during the provider migration. Please extract a shared route projection helper, or use the real provider registry here.
Collaborator
Author
There was a problem hiding this comment.
Partly fixed, partly deferred. The display name and plan command now come from the real provider registry (createTestProviderRegistry(), which loads the first-party plugin declarations and runs them through buildPluginProviderRegistration) via the same resolveProviderPlanCommand the route calls — the row snapshots came out byte-identical, which confirms the old literals matched. The event budget and inline-output limit were already read from defaultFeatureFlags / DEFAULT_MAX_INLINE_OUTPUT_CHARS. Extracting a shared route projection helper is a production refactor, and this PR is deliberately test-only; the contract PR that follows is the right place for it.
| // BB_PROVIDER_CORPUS_DIR and write snapshots next to it, so the task is | ||
| // uncacheable and must see those two variables (strict env mode strips | ||
| // everything undeclared). Without the variable the suites skip. | ||
| "@bb/server#test:provider-corpus": { |
There was a problem hiding this comment.
🚨 slopcop/review — This task key and its comment already appear above.
JSONC accepts the duplicate, but the last value silently wins. This creates configuration risk if one copy changes later. Please remove one complete block.
Collaborator
Author
There was a problem hiding this comment.
Good catch — a botched re-apply after a Prettier revert. The duplicate block is gone; turbo.json has one @bb/server#test:provider-corpus task.
There was a problem hiding this comment.
🚨 SLOP COP 🚨 · review
Plain English summary
This pull request adds regression tests for provider timelines and permission rules. It uses private conversation data to save expected timeline rows. It also measures timeline cost and checks every permission-policy combination. The private data stays outside the repository.
Review findings
I found eight issues. The performance gate needs correction before it can provide a reliable baseline.
- High: The fixed 750 ms limit failed one of three unchanged runs.
- High: The calibration uses the same timeline path, so a common regression can cancel from the ratio.
- Medium: The ratio uses two independent minimum values instead of paired samples.
- Medium: Compare mode does not check the saved sample count or calibration size.
- Low security: An invalid provider segment can move snapshot access outside the corpus directory.
- Medium: The loader does not validate identities across the manifest, metadata, and event rows.
- Medium: The harness copies route policy and can drift from the production route.
- Low:
turbo.jsondefines the same provider-corpus task twice.
I left a line comment for each issue. I did not approve this pull request or request changes.
Checks
- Type checks passed for
@bb/test-helpers,@bb/agent-runtime, and@bb/server. - The focused runtime permission matrix passed 82 tests.
- The focused server permission matrix passed 65 tests.
- A separate full runtime pass completed 504 tests.
- The performance check passed twice and failed once on the unchanged head.
- The private corpus was absent, so the private baseline suites did not run.
- A Doobie smoke test loaded the dev app home page at the exact pull request SHA.
- The pull request changes no product route, so no changed route existed for a deeper browser test.
Collaborator
Author
|
Coordinator review — APPROVE (pending CI green; SlopCop is now disabled so this review is the gate).
This is the regression oracle the bridge/projection PRs depend on. Good to merge once CI is green — Sawyer merges (I don't). WS1b/WS3 will consume the harness.
|
Collaborator
Author
|
Harness tuning finding from running this oracle against WS1a (#2136): the Not blocking this PR (the gate design is sound and it correctly didn't flag a real regression). Suggested follow-up for whoever next touches it: widen
|
SawyerHood
force-pushed
the
bb/provider-baselines
branch
from
August 21, 2026 09:26
4af0367 to
bbfa206
Compare
SawyerHood
changed the base branch from
main
to
bb/ws1a-assembler-testing-kit-stack-on-2124-thr_jkbj56vr97
August 21, 2026 09:26
Collaborator
Author
|
Coordinator re-review after the re-stack onto WS1a — APPROVE (pending CI on This was a landing blocker I found: the matrix tests imported I verified the A5 pin held exactly, by diffing the literal outcome map between the old head ( Bonus signal worth recording: this is the third independent A4 confirmation — all 307 Landing slot: after #2136 WS1a (and after #2148 WS2a, which also sits on WS1a — rebase onto #2148's head before landing so the stack stays linear). Sawyer merges; I do not.
|
Collaborator
Author
|
CI on
Neither file is touched by this PR. On WS1a's own green run (32455240867) the same two tests took 3,497 ms and 3,159 ms and the lifecycle file took 40 s, so those tests carry ~30% headroom and tip over on a slower runner. Locally on this head the two files pass alongside the new matrix file 3/3 (108 tests, ~5 s total). The matrix test itself spawns one Recommendation for WS1a (#2136): give the bridge-spawning lifecycle/multi-thread tests an explicit timeout (15 s) or set |
SawyerHood
force-pushed
the
bb/provider-baselines
branch
from
August 21, 2026 09:44
bbfa206 to
e8ecfab
Compare
SawyerHood
force-pushed
the
bb/provider-baselines
branch
from
August 21, 2026 10:33
e8ecfab to
180439b
Compare
SawyerHood
changed the title
SawyerHood
changed the base branch from
bb/ws1a-assembler-testing-kit-stack-on-2124-thr_jkbj56vr97
to
bb/ws2a-registry-providerinfo-stack-on-2124-thr_hbi4kggyzb
August 21, 2026 10:33
Collaborator
Author
|
CI on
|
Reads BB_PROVIDER_CORPUS_DIR (manifest.json, threads/<provider>/<id>/
{meta.json,events.ndjson}) and decodes event rows with the domain
stored-event parser the server uses. Tests guard on corpusAvailable();
the corpus itself stays out of the repo.
Co-Authored-By: Claude <noreply@anthropic.com>
Loads each corpus thread into in-memory SQLite with its original ids and projects every timeline page the way the timeline route does (default and nested variants). BB_PROVIDER_CORPUS_SNAPSHOT=write mints the baseline under $BB_PROVIDER_CORPUS_DIR/snapshots/rows; the default compare mode fails on any diff that snapshots/allowlist.json does not cover and on allowlist entries that cover nothing. Skips when the corpus is absent. Co-Authored-By: Claude <noreply@anthropic.com>
Measures the 10 largest corpus threads per provider: latest page and full page walk, five profiled builds each, interleaved with a synthetic calibration thread. The gate compares the normalized minimum (thread min over calibration min) against perf-baseline.json with a 10% budget, and the median persisted event size with a 15% budget. A CI micro-benchmark walks a synthetic 10k-event thread that covers every item kind and needs no corpus. Co-Authored-By: Claude <noreply@anthropic.com>
Runtime: every (permission policy × approval subject × approvalEnforcedBy × deny availability) cell of handleRuntimeProviderRequest as a literal table, exhaustive at the type level against the domain unions. Server: escalation by turn initiator and thread shape, and the five accepted runtime permission policy shapes out of the 54-cell cross product. Co-Authored-By: Claude <noreply@anthropic.com>
scripts/provider-corpus/snapshot-rows.sh wraps the new uncacheable @bb/server#test:provider-corpus turbo task, which passes the corpus variables through strict env mode. docs/debugging-and-qa.md documents the corpus, the gates, the allowlist format, and how to refresh a baseline. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
… files, drop duplicate turbo task - turbo.json: remove the duplicated @bb/server#test:provider-corpus block. - Perf gate: calibrate with a fixed JSON/sort workload that shares no code with the timeline, so a uniform build regression still moves the ratio; gate on min build / min calibration over interleaved samples; record the paired-ratio minimum as a diagnostic; write mode keeps the median of three attempts; compare mode refuses a baseline written with other settings. - Synthetic benchmark: gate on the minimum of five walks under 1,500 ms. - Corpus reader: thread and provider ids must be one safe path segment; loadCorpusThread resolves through the manifest and checks that meta.json and every event row agree on id, provider, reasons, and row count. - Harness: display name and plan command come from the real provider registry and resolveProviderPlanCommand; snapshot paths are confined to snapshots/rows. Co-Authored-By: Claude <noreply@anthropic.com>
… shape WS1a deleted the ProviderAdapter interface and the fake adapter. The runtime matrix now drives handleRuntimeProviderRequest through the real bridge-protocol adapter built for the scripted echo launch, with the initialize handshake setting approvalEnforcedBy and a canonical interaction/request on the wire. The 80 pinned outcomes are unchanged; the tool_use subject the v3 contract added gets its own 20 measured cells, as the type-level union guard required. The server matrix follows resolvePermissionEscalation, which now takes only the initiator, and pins the reviewer vocabulary to the policy union since the domain no longer exports the list. The corpus harness derives the three types the dead-code sweep un-exported instead of re-exporting production code. Co-Authored-By: Claude <noreply@anthropic.com>
SawyerHood
force-pushed
the
bb/provider-baselines
branch
from
August 21, 2026 10:48
180439b to
8d2c46c
Compare
SawyerHood
added a commit
that referenced
this pull request
Aug 21, 2026
…2136) Stacked on #2124 (`bb/provider-contract`). WS1a of the provider-plugin migration: the generic assembler, one streaming + one usage dialect, extension ingest validation, the published testing kit, the scripted echo bridge as the harness default, and — last commit — the deletion of the legacy `ProviderAdapter` path and the v2 delta dialects. **Do not merge.** Coordinator reviews, Sawyer merges the stack. ## What was wrong The contract PR landed the grammar v3 vocabulary but left the assembler stubbed (`UnsupportedDeltaShapeError` in every v3 shape and in `extension.state`), kept two streaming and two usage dialects, validated no extension payload, published no testing kit, and the runtime still carried a legacy `ProviderAdapter` interface whose only non-bridge implementation was a 674-line fake adapter driven by a legacy-dialect script (the integration harness's default provider). ## What changed (one commit per layer; the last deletes) 1. **Assembler builds the v3 core kinds** (`92ae9cd`) — `fileRead`, `search`, `planSteps` open pending and settle from the terminal shape like `command`; a foreground `delegation` settles through `item/completed`, a `background: true` delegation is thread-attached like a background task (`item/delegation/progress|completed`, no turn needed, survives turn settlement and `session.ended`). `ASSEMBLER_GRAMMAR_VERSIONS` → `[2, 3]`. 2. Presentation persistence shipped upstream in #2124 (`fc88906`); nothing to do here. 3. **Extension kinds + ingest validation** (`f174c5d`) — `extension` items and the new `thread/extensionState/updated` event assemble. The server validates every extension payload against the owning plugin's declared Standard Schema at ingest (`apps/server/src/internal/extension-payloads.ts`; registrations carry the validators, the registry resolves `"<pluginId>/<name>"` through the plugin-id prefix; 64 KiB cap). An undeclared kind, a schema miss, a validator error, or an oversized payload is persisted as `provider/unhandled` in the same batch slot — G11-visible, never dropped, never stored unvalidated. `extensionKindSchema` parses to the `ExtensionKind` type. 4. **One streaming dialect, one usage dialect** (`cce9415`) — every text stream is an item keyed like any other: `item.textDelta`/`item.textClose`, anonymous streams keyed by `key.channel` (+ `parentRef`). `usage { total, last, modelContextWindow }` is forwarded verbatim; bridges that report per turn (claude, pi) accumulate with the bridge kit's `addTokenUsage` and reset at `session.reset`; codex sends `contextWindow` (now with `providerTurnId`) beside it. All four bridges + the echo example migrated. **Calibration goldens unchanged** for codex, claude, acp, pi. 5. **`provider/recovery`** (`e5f5a3b`) — decoded by the adapter, forwarded to the runtime's new `onProviderRecovery` hook (the daemon logs it; WS4 acts per kind). Grammar negotiation shipped upstream (`0816b4c`). 6. **Published testing kit** (`03815e3`) — `@get-bb/plugin-sdk/provider-bridge/testing`: conformance kit, the real assembler, delta→event collector, JSON-RPC harness, calibration normalizer. Framework-agnostic (`captureBridgeJsonRpcOutput` patches `process.stdout.write`, no `vi`). `experimental_` value names + `docs/api_to_audit.md` entry; G10 doc-sync test asserts the entry. The assembler moved into `@bb/provider-bridge-protocol` (`assembler` subpath) — the SDK cannot depend on the runtime (cycle). The echo example and every first-party bridge suite import only `@get-bb/plugin-sdk/provider-bridge` + the testing entry; the echo example's `@bb/*` devDependencies are gone. **Scripted echo bridge as the harness default** (`3d3cb9e`) — `tests/scripted-echo-provider` (the echo bridge + scripted directives: `delay:`, `approve:`, `ask_user`, `call_tool:`, `hold_turn`, `fail_turn:`, …; session/process behaviour via `providerOptions.scripted` / `SCRIPTED_ECHO_OPTIONS`; `SCRIPTED_ECHO_RECORD_PATH` records every request, `SCRIPTED_ECHO_PROCESS_LOG_PATH` every process step). Passes the conformance suite. The integration harness has no `adapterFactory` seam: the fake providers are declarations backed by the built scripted artifact, run by the daemon through the real adapter. 7. **Deletion** (`a9a3950`) — `ProviderAdapter` → concrete `BridgeProtocolAdapter`; `adapterFactory` / `createAgentRuntimeWithAdapters`; the fake adapter + script; `message.delta/close`, `usage.turn/exact` from schema + assembler; assembler `[3, 3]` and every bridge reports it (a bridge that predates `grammarVersions` reads as v2 and is refused at the handshake; the conformance handshake scenario checks the same). Runtime unit suites + the daemon's thread.stop race suite run the scripted echo bridge through the real bootstrap + adapter + assembler; the process manager gains a `createAdapter` seam for raw-script spawn/stderr/exit tests. `HOST_DAEMON_PROTOCOL_VERSION` → 148. ### Tests deleted (subject no longer exists) - command-contract: "rejects required adapter commands that return no-op plans", "rejects no-op steer commands", the noop half of "rejects no-op stop commands" (only the fake adapter's `buildCommandPlan` seam could plan a noop; handshake gating is pinned in `bridge-protocol-adapter.test.ts`). - lifecycle: "passes Codex-shaped thread/start ids to accepted command translation" (`translateAcceptedCommand` is a no-op for bridges), "preserves merged shell env when reconfiguring a thread" (the session-rebuild path is unreachable: bridges classify every settings change as `live`; env is covered by the start/resume tests), "drops a delta into an item nothing opened, with a visible warning" (the only seam that could feed a malformed event was the `translateEvent` override; the grammar gate is exercised by the new replayed-turn bridge test). - input-accepted: "suppresses provider-emitted user message echoes" (bridges never emit `userMessage`). - multi-thread: "maps thread/started before identity", "drops unscoped provider events" (legacy `thread/event` dialect routing; every `thread/delta` names its bb thread id). - process-lifecycle: "continues startup when an optional post-initialize read is unsupported" (the only post-initialize request is the handshake itself). Everything else is ported faithfully; literal-id assertions became assertions on the assembler-minted ids (#1224), `AdapterCommand` recordings became request-record assertions on the same wire facts. ## Regression oracle status - **Parity replay (A2)**: `bb/provider-recordings` does not exist on origin; not in the base. Coordinator requires it before merge. - **Corpus row snapshots (A4, #2121)**: not in the base. - **G1 ratchet (#2120)**: not in the base. This PR adds no provider-id literal to core (the scripted bridge's codex-shaped archived error is test-only). - **Conformance kit**: green for echo, scripted echo, codex, claude-code, acp, pi. - **Calibration goldens**: byte-identical for codex, claude, acp, pi through the dialect migration. ### Intended byte-level difference (allowlist) - **WS1a #2136, layer 4**: a provider-named text item that streamed before `session.ended` now settles with its accumulated text instead of its opened (empty) shape. Reason: one streaming dialect means the assembler owns the stream text for named items too; losing streamed text on interrupt was the v2 behaviour, not a feature. ### Wire discipline `HOST_DAEMON_PROTOCOL_VERSION` 147 → 148: the daemon emits a new event type (`thread/extensionState/updated`) and its bridges speak grammar v3 only, which a 147 daemon would refuse at the handshake. `PROVIDER_BRIDGE_PROTOCOL_VERSION` stays 2 (envelope and methods unchanged; the grammar range gates). ## Gates (all `--concurrency 4`) Typecheck — green, 27 tasks: `@bb/domain @bb/provider-bridge-protocol @bb/agent-runtime @bb/server @bb/host-daemon @bb/host-daemon-contract @get-bb/plugin-sdk @bb/thread-view @bb/db` and `provider-codex provider-claude-code provider-acp provider-pi echo-provider scripted-echo-provider @bb/integration-tests @bb/app @bb/mobile @bb/cli`. Tests: | package | result | |---|---| | @bb/domain | 148/148 | | @bb/provider-bridge-protocol (incl. the assembler suite) | 213/213 | | @bb/host-daemon-contract | 52/52 | | @get-bb/plugin-sdk | 127/127 | | @bb/thread-view | 379/379 | | @bb/db | 406/406 | | @bb/agent-runtime (incl. pi conformance) | 336/336 | | bb-plugin-provider-codex | 172/172 | | bb-plugin-provider-claude-code | 263/263 | | bb-plugin-provider-acp | 181/181 | | bb-plugin-echo-provider / scripted-echo-provider | 2/2, 1/1 | | @bb/host-daemon | 556/556 | | @bb/integration-tests (fake stack on the scripted bridge) | 55/55 | | @bb/server | 1821/1823 — two pre-existing, environmental locals: `internal-skill-trees` (umask 0664 vs 0644, passes in CI) and `plugin-update` "waits one full interval" (5 s timeout under load; test and subject untouched by this PR) | Perf (assembler micro-benchmark, 20 000 mixed turns = 1 340 000 deltas → 1 380 000 events, 3 runs each, same workload): contract head (v2 dialect) min 457 ms / 2.93 M deltas/s; this branch (v3 dialect) min 436 ms / 3.07 M deltas/s — ~4 % faster, heap delta no worse. Within the +10 % gate. > AGENT GENERATED: by Claude Opus 5 --------- Co-authored-by: Claude <noreply@anthropic.com>
SawyerHood
added a commit
that referenced
this pull request
Aug 21, 2026
…rdings, and the parity harness (#2153) Stacked on #2136 (WS1a, `bb/ws1a-assembler-testing-kit-stack-on-2124-thr_jkbj56vr97`), itself stacked on the contract #2124. Landing order: contract → WS1a → this → WS2a → #2121 → codex → claude. **Why stacked, not independent.** `@bb/provider-parity` and the three `bridge.recorded-conformance.test.ts` files import `createBridgeDeltaEventCollector`, which lives at `@bb/agent-runtime/test/bridge-delta-assembly` on `main` and at `@bb/provider-bridge-protocol/testing` on WS1a. No ordering of two independent PRs yields a buildable `main`, so this PR follows the move (and drops `provider-parity`'s `@bb/agent-runtime` dependency). The recorded `bridge→runtime` lanes are grammar v2, which WS1a's v3-only assembler refuses, so every replayable cell now also carries a `bridge→runtime.current.ndjson` written by `pnpm --filter @bb/provider-parity rerecord --plan-with <main checkout>` through this branch's bridges; they assemble to the **unchanged** pins in `row-counts.json`. The 10 pi cells (in-process SDK, not replayable) keep their pins and are reported, not failed. **Do not merge.** Coordinator reviews, Sawyer merges the stack. ## What was wrong The provider-plugin migration will break byte-equivalence with today's translators on purpose, which removes the regression oracle the goldens provide. Calibration sessions are scripted fakes and the integration suite runs a fake adapter, so nothing checks a bridge against what the provider CLIs really emit across the behavior matrix. Design: "Regression confidence" (A2, A3) and "Corpus" in the ideal-state provider API spec. No bridge may migrate before this lands. ## What changed **Bridge record mode** (`BB_PROVIDER_BRIDGE_RECORD_DIR`, off by default, additive) - `bridge-worker-entry.ts` tees both sides of the runtime wire for every bridge, first- or third-party, before the bridge module loads. - Bridges tee their provider child with `experimental_recordProviderChildIo(child, { threadId })` (codex `app-server-connection.ts`, acp `agent-connection.ts`). Claude's pipe belongs to the Agent SDK, so `sdk-session.ts` takes the SDK's `spawnClaudeCodeProcess` seam over when `experimental_isProviderBridgeRecording()`. Pi runs in-process and records its SDK event boundary (`AgentSessionEvent` in, prompt/abort/compact out). - Layout `<dir>/<providerId>/<threadId>/<direction>.ndjson` (`_process` for lines that belong to no thread); one `{ts, run, seq, dir, line}` per line, appended per line, never buffered. Responses land in the scope of the request they answer. `run` identifies the bridge process so a thread served by several processes merges back in order. - The daemon forwards the variable from its own env (not the shell env, which doubles as the agent's shell environment); the runtime appends the provider id. `withoutBridgeRuntimeEnv` and the existing `BB_*` allowlist both strip it from provider children. - Docs: `docs/provider-bridge-protocol.md` ("Record mode"), `docs/debugging-and-qa.md`, `docs/configuration.md`, `docs/api_to_audit.md` (both new `experimental_` SDK members). **Recordings** — 52 redacted cells under `packages/provider-bridge-protocol/recordings/<provider>/<cell>/` with a `manifest.json` each (provider, cell, CLI version, date, description, lane line counts). `scripts/provider-recordings/redact.mjs` rewrites paths under `$HOME`, emails, `ls -l` owner columns, token shapes (`bbde_`, `ghp_`, `github_pat_`, `sk-`, `sk-ant-`, `xox?-`, `bbcm_`, JWTs, bearer values, `Authorization` headers), secret-shaped env keys, collapses Claude's `system/init` and `control_response` catalogs to names, and trims strings over 2,000 chars to head/tail with a marker. It is idempotent and exits 3 if any pattern survives. `package-cells.mjs` cuts the raw per-thread recordings into cells. Raw recordings stay in `~/.bb/provider-recordings/raw` (gitignored). **Parity harness (A2)** - `@bb/provider-bridge-protocol/testing/parity`: `replayRecording` drives the recorded `runtime→bridge` lane into a bridge process and `replay-provider-child.mjs` plays the provider lanes back as the bridge's child — the recording *is* the script. The child gates each recorded provider line on the live bridge's own writes (method/subtype for requests, id for responses), maps bridge-minted ids to recorded ones, cuts the lanes into one segment per spawned child, paces spontaneous lines behind the harness's cursor so steers and interrupts land between the same two provider lines they did live, and never hangs a divergent bridge (stall → generic answer, logged). Serves JSON-RPC (codex, ACP) and the Claude CLI control protocol. Each replay runs in its own temp workspace and (for Claude) its own `CLAUDE_CONFIG_DIR`, with a seeded source transcript per recorded `thread/fork`, so recordings replay on any machine. - `compareParity(old, new, allowlist)` diffs normalized events (the resurrected `diffCalibrationStreams`, `#2140` had removed it), normalized rows, and grammar drops. Allowlist entries are `{provider|"*", cell|"*", layer, path (JSON pointer with `*`/`**`), pr, reason}`; an entry that masks nothing is reported stale and fails. - `@bb/provider-parity` wires the real delta assembler and `@bb/thread-view` projection (mirroring `timeline.ts`), owns `pnpm parity --old <checkout> --new . [--provider] [--cell] [--dump-dir]`, and loads each leg's assembler and projector from that leg's own checkout (its `@bb/provider-parity`, else the collector at its pre/post-WS1a home), so a `main` `--old` leg assembles with `main`'s assembler. - `parity.self.test.ts` (CI): every cell assembles to the counts pinned in `recordings/row-counts.json` (events, rows; `provider/unhandled` and grammar drops may only go down — G11), every allowlist entry names a PR, reason, and pointer (staleness is judged by `pnpm parity --old <main>`, which an old==new replay cannot), and every replayable cell replays through the current bridge with zero event/row/grammar diffs and zero stalls. `UPDATE_PARITY_ROW_COUNTS=1` rewrites the pins deliberately. **Current bridge lanes (merged from #2177, reviewed as harness owner)** — a recording is never rewritten. When a bridge change alters what the bridge emits, `pnpm --filter @bb/provider-parity rerecord [--plan-with <recording-time checkout>]` writes `bridge→runtime.current.ndjson` beside the recorded lane; the self-suite and recorded conformance pin/compare against it when present, and `pnpm parity` paces the old leg from the recorded lane and the new leg from the current one. Also from #2177: `pnpm parity --dump-dir`, a 300 ms drain before each replayed runtime request and a 50 ms gap after a replayed response (the steer-ack position race seen twice in CI). Two follow-ups on top: the harness's own `initialize` response is kept out of re-recorded lanes, and re-recorded lanes pass through `redact.mjs` before they are written (a bridge error can quote the replay child's command line). 39 current lanes are committed (every replayable cell) — see "Why stacked" above. **Conformance** — `checkRecordedCellReplay` + `replayRecordedCells` add the recorded-traffic scenario set (`recorded/<cell>/{replays, events-schema-valid, grammar, turn-lifecycle, not-empty}`); each first-party bridge gets `bridge.recorded-conformance.test.ts` over turn, steer, stop, approval allow/deny, question, resume, fork. No translation behavior change. No wire change; `HOST_DAEMON_PROTOCOL_VERSION` is untouched. ## How you verified On the stacked head `39066e19b` (WS1a base), one Turbo invocation at a time at `--concurrency 4`: typecheck 12/12 tasks (protocol, parity, agent-runtime, host-daemon, plugin-sdk, codex, claude-code, acp); tests — protocol 218, codex 173, claude-code 262, acp 182 (each `bridge.recorded-conformance.test.ts` green), parity self-suite 43/43 with `row-counts.json` untouched; `pnpm parity --old . --new .` 39 passed / 0 failed / 13 skipped; `pnpm rerecord --plan-with /home/sawyer/projects/bb` 39 OK / 0 STALL; redaction sweep over recordings + current lanes `0 survivors`, idempotent. Earlier, on the `main`-based head `0923c1c59` (after merging #2177): CI green (Checks, Package Smoke ×2, Tests app-1/2/3, integration, packages, server); 21/21 Turbo tasks, 1,848 tests (protocol 101, parity 43, agent-runtime 418, host-daemon 552, plugin-sdk 117, codex 173, claude-code 262, acp 182); the steer/stop-interrupt cells stable across 3 repeated runs. New tests in this PR: recorder routing/tee/oversize, worker-entry record tee, `withoutBridgeRuntimeEnv` strips the knob, daemon forwards it to bridge processes but not the shell env, the 43-test parity self-suite, three recorded-conformance suites. An earlier CI run on `main` failed in `packages` because ACP and Claude replays depended on the recording machine (`cwd`, `~/.claude` transcripts for `forkSession`, `PATH`); reproduced locally with the directory moved away and a bare `HOME`, fixed as described above. **Matrix** (`codex-cli 0.149.0`, `Claude Code 2.1.238` / Agent SDK 0.3.197, `cursor-agent 2026.08.11`, pi `@earendil-works/pi-coding-agent 0.84.0`, recorded 2026-08-21 through `scripts/bb-dev-app` + the `bb` CLI): | cell | codex | claude-code | acp-cursor | pi | | --- | --- | --- | --- | --- | | 1 turn (shell + edit + read) | ✅ | ✅ | ✅ | ✅ | | 2 steer mid-turn | ✅ | ✅ | ✅ | ✅ | | 3 stop mid-turn, new turn | ✅ | ✅ | ✅ | ✅ | | 4 approval allow | ✅ | ✅ | ✅ | ⛔ pi has only `full` mode; nothing asks | | 4 approval deny | ✅ | ✅ | ✅ | ⛔ same | | 5 user question | ✅ (bb tool) | ✅ (`AskUserQuestion`) | ✅ (bb tool) | ✅ (bb tool) | | 6 subagent / delegation | ✅ (native `subAgentActivity`) | ✅ (`Agent`) | ✅ (bb delegation) | ✅ (bb delegation) | | 7 resume after `thread stop` | ✅ | ✅ | ✅ | ✅ | | 8 fork | ✅ | ✅ | ✅ recorded as the agent's refusal: `cursor-agent` advertises no `session/fork` (0 events, pinned) | ✅ | | 9 plan mode | ✅ (`/plan` command mention) | ✅ (`claudeCodePermissionMode: plan`, plan approval interaction) | ⛔ no plan command | ⛔ no plan command | | 10 model list | ✅ (`_process` scope) | ✅ | ✅ | ✅ | | 11 web search / fetch | ✅ (`webSearch` item) | ✅ (`WebSearch` + `WebFetch`) | ✅ (Web Fetch tool) | ✅ (curl fallback; pi has no web tool) | | 12 compaction | ✅ | ✅ | ⛔ server 409: provider does not support manual compaction | ✅ | | 13 archived session resume | ✅ (archived natively via app-server `thread/archive`; bridge unarchives and retries) | n/a | n/a | n/a | | 13 empty rollout | ✅ (0-byte rollout → `failed to read session metadata`) + bonus `missing-rollout` | n/a | n/a | n/a | | 13 auth failure | ✅ real 401 ×11 via empty `CODEX_HOME` | ✅ "Not logged in" via empty `CLAUDE_CONFIG_DIR` | ⛔ not attempted: no config-dir knob to point at an empty store without touching the real login | ⛔ not attempted: pi's keys live in its own config; no safe override | | 13 429 | none occurred naturally (not forced) | — | — | — | **Redaction sweep** (`node scripts/provider-recordings/redact.mjs packages/provider-bridge-protocol/recordings /tmp/redact-stack`): ``` redacted 241 recording files into /tmp/redact-stack (home=/home/sawyer → /home/user); 0 survivors ``` `diff -rq` between the committed fixtures (recordings + current lanes) and that output is empty (idempotent). The only remaining occurrence of the username is the public skill name `bb-global-skills:sawyer-voice` in Claude's advertised skill catalog. **Parity self-run** (`pnpm parity --old . --new .`): `39 passed, 0 failed, 13 skipped (52 cells)` — the 13 skips are the 3 process-scoped `model-list` cells (no thread events) and the 10 pi cells (in-process SDK, no child to replay; still pinned and assembled by the self-suite). Every PASS line reports identical old/new counts, e.g. `codex/steer: old 87 events/3 rows, new 87 events/3 rows, unhandled 0→0, grammar drops 0→0`. The cross-checkout path was exercised too: `--old /home/sawyer/projects/bb` (on `main`, no `provider-parity` package) loaded that checkout's `@bb/agent-runtime` collector and bridge and matched on all 16 codex thread cells. **Sizes**: recordings 6.9 MB total including the 39 current lanes (6.2 MB without) — codex 1.6 MB, claude-code 1.2 MB, acp-cursor 932 KB, pi 2.5 MB recorded; largest cell `pi/turn-tools` 828 KB. Fixes none (step-0 PR of the provider-plugin migration). > AGENT GENERATED: by Claude Opus 5 --------- Co-authored-by: Claude <noreply@anthropic.com>
SawyerHood
added a commit
that referenced
this pull request
Aug 21, 2026
…bridge to grammar v3 with presentation (#2164) Stacked on #2121 (corpus harness + permission matrix, `bb/provider-baselines`) ← #2148 (WS2a) ← #2153 (recordings) ← #2136 (WS1a) ← #2124 (contract). WS1b-codex of the provider-plugin migration: the codex bridge speaks grammar v3 with a presentation on every item, its natives map to the core kinds, bb-injected tools carry their presentation, goals and the macOS permission profile are codex extension kinds, and the codex v2 path is deleted. **Do not merge.** Coordinator reviews, Sawyer merges the stack. ## Stack contract #2124 → WS1a #2136 → recordings #2153 (incl. #2177 and the `PARITY_INITIALIZE_ID` follow-up) → WS2a #2148 → corpus harness #2121 → **this PR** → WS1b-claude #2178. Six commits: the five codex layers below plus one `parity:` commit carrying this PR's allowlist entries and the codex `bridge→runtime.current.ndjson` lanes re-recorded against this bridge with #2153's `pnpm rerecord --provider codex --plan-with <main checkout>`. No vendored harness copies remain (they were needed only while #2153 was outside the stack). The A4 corpus check was run from a throwaway worktree with #2121's harness cherry-picked on top, since #2121 is not in the stack. ## What was wrong The codex bridge emitted v2-shaped items: no presentation (core thread-view kept codex's tool-name tables), native sub-agents as `tool` items named `spawnAgent`, `update_plan` as a turn-level event the UI discards, goals as core `thread/goal/*` events, bb-injected tools without `server`, and a macOS permission profile on a command approval failed the whole approval. Open work rode an out-of-band `thread/openWork` notification. ## What changed (one commit per layer; the last deletes) 1. **Presentation on every item.open/close** — `plugins/provider-codex/src/presentation.ts` is the one place codex tool-name knowledge lives: shell commands (wrapper stripped from the headline), file edits, the bundled `node_repl` server ("Ran JavaScript" with the call's title), other MCP servers by tool name, dynamic tools, collab verbs, web search/fetch, image views, reasoning, messages, plans, compactions, the synthesized sub-agent spawn. An invariant suite drives one item per codex native through the real translator and asserts every lifecycle delta carries one. 2. **Delegation + planSteps** — the synthesized native sub-agent is a foreground `delegation` (`childRef` = agent thread id, `label` = agentPath); a follow-up to a settled agent re-opens the same item and the row closes only when the agent owes nothing more (the exact open-work predicate); a dead app-server child settles its open delegations as failed on the wire. A collab call that names its receiver is a delegation to it; a bare `wait` stays a tool item with its collab presentation. `turn/plan/updated` becomes a settled `planSteps` snapshot per update. `RuntimeBackgroundWorkState` counts a pending delegation as open work. **Thread-view projects a `delegation` item to the existing delegation row with the child content nested** — without this the row and every child message under it vanish before the projection workstream lands (the projection suppresses orphans with a `parentToolCallId`). `planSteps` items are not projected yet (status quo: codex plans were discarded). 3. **bb-injected tools carry their presentation** (Q31) — `bb.agents.registerTool({ experimental_presentation })` (+ `docs/api_to_audit.md`); the server resolves one presentation per tool at its boundary (declaration → status labels → generic label + the plugin's branding glyph / `Toolbox`) onto `DynamicTool.presentation`; the codex bridge emits calls to injected tools as `{ server: "bb", tool }` with that presentation. ask-user-question and workflows declare theirs (AskUserQuestion and `bb_workflow_result` collapse by default). `HOST_DAEMON_PROTOCOL_VERSION` 149 → 150 (149 is WS2a's #2148, this PR's base; the history reads 147 → 148 → 149 → 150). The field is optional and `dynamicToolSchema` is not strict, so an older daemon strips it and keeps working; the bump follows the repository rule for a widened server↔daemon wire. Optional per A1: every committed recording's runtime lane predates the field and must keep replaying; the stabilization pass makes it required. 4. **Goals + macOS profile as codex extension kinds** — `provider-codex/goal` (state; `null` once cleared) and `provider-codex/macos-permission` (item) declared on the registration with plugin-owned zod schemas (`extension-kinds.ts`). The bridge emits goals as extension state. A command approval asking for macOS capabilities now reaches the user for the command; the profile rides the timeline as its own row saying bb cannot grant it (the plugin-rendered approval is WS5's). **Read-time conversion in core:** `parseStoredThreadEvent` decodes persisted `thread/goal/updated|cleared` rows into the extension state (the one path every stored-event read takes); thread-view goal extraction and the runtime's goal-clear wait read that state. The sidebar's latest-goal query becomes a latest-thread-state-by-kind query (partial index widened to `thread/extensionState/updated`, migration **0106** `thread_state_index` — regenerated with Drizzle so its snapshot chains from WS2a's 0105 `provider_settings_to_plugins`; `json_extract` kind filter). Verified against the corpus: the 721-row goal thread renders its goal unchanged. 5. **Delete the codex v2 path** — `thread.goal`/`thread.goalCleared` leave the grammar and assembler (G3 snapshot updated; `PROVIDER_BRIDGE_PROTOCOL_VERSION` stays 2 under the grammar range, as WS1a's v2 deletion did); the `thread/openWork` notification leaves the protocol, adapter and reaper (an unknown notification is ignored); the bridge drops its open-work reporting and the hard-coded AskUserQuestion presentation. Kept: the bridge's knowledge of its spawn/resume collab verbs for a receiver-less call — tool-name knowledge in the bridge is the point, not a remnant. The `thread/goal/*` domain event types remain as read-only legacy vocabulary. Naming note: the spec says `codex/goal`; the extension-kind namespace is the **plugin id**, which is `provider-codex`, and that is how the registry resolves the schema. ## Regression oracle All turbo invocations with `--concurrency 4`; perf suite ignored (known-noisy, flagged on #2121). - **Conformance**: codex scripted suite + recorded conformance over all 17 recorded cells (incl. archived-resume, auth-failure, empty-rollout, missing-rollout) green. - **Parity (A2)**, `pnpm parity --old <origin/main worktree at f6fb434> --new . --provider codex`: **16 passed, 0 failed, 1 skipped** (process-scoped `model-list`). Event and row counts equal in every cell. `recordings/parity-allowlist.json` names every intended byte-inequivalence with `#2164` and a reason — 26 entries, four classes: `presentation` on items (15 cells), `server: "bb"` on the AskUserQuestion call (user-question), the sub-agent spawn as a `delegation` item + its row (`subagent`, events `/6` `/28`, rows `/0/children/0/{toolName,output}`), and `thread/goal/cleared` → `provider-codex/goal` extension state (one event index in each of 6 goal-bearing cells). Zero unlisted diffs, zero stale entries. claude-code **13/13** and acp-cursor **10/10** replay against main with **zero diffs** (no entries). - **Corpus (A4)**, 307 threads / 93,262 rows: **zero diffs**, claude-code and codex alike; no corpus allowlist entry was needed (the only read-time change, goals, projects identically by design). - **G11**: `provider/unhandled` flat on every codex cell (0→0; auth-failure 1→1) and on the corpus. - **G1**: 209 → 209 (no provider-id literal added or removed in core; `"provider-codex/goal"` is not a provider-id literal by the ratchet's regex). - **Parity self-suite** (#2153's, in the stack): 43/43 with the codex current lanes re-recorded against this bridge; row-count pins **unchanged** (every current lane assembles to exactly the pinned counts); the claude/acp current lanes #2153 produced on WS1a's bridges hold as they are. Tests on the final stack (all forced, `--concurrency 4`, one turbo at a time): db 406 (migration chain 0104 → 0105 → 0106 incl. replay-on-existing-DB cases), agent-runtime 337, codex 193 (scripted + recorded conformance), host-daemon-contract 52, provider-parity 43; typecheck green for codex, agent-runtime, provider-parity, server, db. Earlier full sweep before the re-stacks: domain 150, thread-view 382, provider-bridge-protocol 218, plugin-sdk 127, host-daemon 552, integration 55, claude-code 262, acp 182, workflows 223, ask-user-question 36, scripted echo 1; server 1828/1829 (the one failure is the known local umask case in `internal-skill-trees`, passes in CI); typecheck green across 25 packages incl. app, mobile, cli. ## Not done - **Live QA cells via bb-dev-app** (turn/steer/stop/approve/deny/question/subagent/resume/fork/plan with screenshots) — not run; the coordinator schedules them separately. - `planSteps` rows are not projected by thread-view (no regression: codex plans were discarded before); the projection workstream owns it. - The plugin-rendered macOS approval (the profile round-trips only as a visible row today) — WS5. > AGENT GENERATED: by Claude Opus 5 --------- Co-authored-by: Claude <noreply@anthropic.com>
SawyerHood
added a commit
that referenced
this pull request
Aug 21, 2026
… presentation (#2178) Stacked on #2164 (WS1b-codex, `bb/ws1b-codex-codex-bridge-to-v3-stack-on-2136-thr_xnid5ftd87`), which sits on WS2a #2148 → the corpus harness #2121 → the recordings harness #2153 → WS1a #2136 → the contract #2124 (SDK 0.4.11 at the bottom). WS1b-claude of the provider-plugin migration: the Claude Code bridge speaks grammar v3 with a presentation on every item, Claude's tool-name knowledge moves out of core thread-view into the bridge, and the bridge's last Claude-specific result structure is deleted. **Do not merge.** Coordinator reviews, Sawyer merges the stack. Stacked on codex rather than WS1a on purpose: codex added the minimal thread-view projection of `delegation` items (row + nested child content) that Claude's `Agent`/`Task` sub-agents need. The recordings + parity harness beneath this PR is the real #2153 (in the chain, with `pnpm rerecord` and the `PARITY_INITIALIZE_ID` fix); the one harness file this PR touches is `redact.mjs` (bare-message ndjson, dash-encoded home paths). The corpus harness #2121 is in the chain too; the A4 run below is native to this tree. ## What was wrong The Claude bridge emitted v2-shaped items: no presentation (core thread-view kept Claude's tool-name tables — `Read`/`Grep`/`Glob` intents, the `Agent`/`Task` delegation row, the TodoWrite/Task* suppression list and todo reducer), `Read` as an opaque `tool` item (the top generic tool in the production corpus: 7,568 calls), `Agent` as a `tool` item named Agent, TodoWrite/TaskCreate/TaskUpdate as tool rows a core reducer had to understand, bb-injected tools as `mcp__bb-bridge__<name>` without `server`, and a structured task-tool `result` that core parsed — and which never matched in production, because the persisted result was the SDK's friendly string, so the task banner did not work for the Task tools. ## What changed (one commit per layer; the last deletes) 0. **Transcript → SDK-stream converter and fixtures** (the spec's first task). `scripts/provider-recordings/convert-claude-transcript.mjs` turns a `~/.claude/projects` session (plus its `<session>/subagents/agent-*.jsonl` sidechains, interleaved by timestamp with `parent_tool_use_id` from the subagent's `toolUseId`) into the SDK stream the bridge would have seen. A transcript has no `result`, no `system/init` and no `task_*` family, so the converter synthesizes them deterministically: a `result` per segment (a root message that stops with a non-tool-use reason, the next prompt, EOF), `task_started`/`task_updated`/`task_notification` for Agent calls from the call, its result (`async_launched` ⇒ backgrounded) and the `<task-notification>` resume, `api_retry`/model-fallback/`compact_boundary` from the system records. Human prompts only delimit turns — a live stream never echoes them (verified against every committed recording); CLI-injected user messages (isMeta context, task notifications) do stream and are kept. `convert-claude-transcripts-sample.sh` rebuilds the committed sample: **12 sessions/windows, 1,076 messages (331 sidechain records), 1.85 MB after redaction**, all from the owner's own corpus threads: plan mode + AskUserQuestion + ExitPlanMode, WebSearch, WebFetch + Read, Edit/Write, foreground and backgrounded Agents with their sidechains, TaskCreate/TaskUpdate + `model_refusal_fallback`, Workflow + Monitor + TaskStop, TaskOutput, ScheduleWakeup, SendMessage, `api_retry`, `compact_boundary`, `mcp__bb-bridge__` tools. Grep, Glob, TodoWrite, MultiEdit and NotebookEdit appear in none of the 2,559 local transcripts; those paths are covered by scripted unit tests. `transcript-fixtures.test.ts` drives each fixture through the `sdk/message` envelope into a real assembler and checks structural invariants (every tool_use opens an item its tool_result settles, sidechain items nest under the spawning call, every turn settles, nothing left open, every started item presented) plus a pinned projection per fixture in `expected.json` (item kinds, tool names, plan snapshots, `provider/unhandled`, which may only go down). 1. **Presentation on every item; the v3 kinds** — `plugins/provider-claude-code/src/presentation.ts` is the one place Claude's tool-name knowledge lives; `tool-classification.ts` maps every tool_use to its shape with that presentation. `Read` → `fileRead`; `Grep` → `search{content}`, `Glob` → `search{path}`; `Edit`/`Write`/`MultiEdit`/`NotebookEdit` → `fileChange` with per-verb labels; `Bash` → `command` (a backgrounded call is labelled as a launch); `WebSearch`/`WebFetch` as before with presentation. `Agent`/`Task` → `delegation` (`childRef` = the call id, which is how the SDK identifies the sub-agent's stream: `parent_tool_use_id`; `background: true` for `run_in_background`, settling at the launch ack on the thread-scoped family; summary = the result text without Claude's `agentId:`/`<usage>` lines; sub-agent type and model in the presentation detail). `TodoWrite` → a collapsed call row plus a settled `planSteps` snapshot from its arguments; `TaskCreate`/`TaskUpdate`/`TaskList`/`TaskGet` → a collapsed call row plus a `planSteps` snapshot of the thread's folded task list (`plan-fold.ts`, reading the SDK's envelope-level `tool_use_result`) after each successful call — channel-keyed close deltas, the latest superseding, the same shape as codex `update_plan`. `ToolSearch`, `TaskOutput`, `Monitor`, `ScheduleWakeup`, `SendMessage`, `AskUserQuestion`, `TodoRead`, `BashOutput` → `tool` with `presentation.suppress`; plan mode, Workflow, TaskStop, Skill, StructuredOutput, worktrees, ListAgents → `tool` with their own labels/glyphs/titles; an unknown tool reads `Running <tool>`/`Ran <tool>`. `mcp__<server>__<tool>` splits into `{ server, tool }`. The compaction item and every close-without-open fallback carry one too; the close re-states the open's. **Thread-view keeps the new kinds rendering** until the presentation-driven projection lands (the same minimal bridge codex added for `delegation`): `fileRead`/`search` items project to the tool row with the intents the legacy Read/Grep/Glob calls produced (tested equal to the legacy rows); a tool call whose presentation says `suppress` is hidden like the legacy name list (failures still render); the todo banner reads a `planSteps` snapshot as-is. No persisted-event projection changed: the legacy tables stay for old rows (G1 unchanged). 2. **Background tasks carry their presentation** — workflows, backgrounded shells and backgrounded sub-agents stay the core `backgroundTask` kind (genericity rule) and say how they read on open/close. Model fallback and `/clear` stay the core events they are. 3. **bb-injected tools** (Q31) — a `mcp__bb-bridge__<name>` call is `{ server: "bb", tool: <bare name> }` with the presentation the server resolved onto the `DynamicTool` definition, learned through `configureInjectedTools` at session construction; a definition without one presents generically under bb's glyph. The server's `statusLabels` enrichment skips items with a server, so nothing relies on it. 4. **Delete the Claude v2 translation path** — the task tools' structured `result` (the shape core's legacy todo reducer read) is gone; the tool row carries the text result like every tool and `planSteps` is the one structured form core sees. The bridge no longer imports the SDK's claude task-tool schemas. Kept on purpose: the bridge's knowledge of its own tool names, and the close-without-open fallback. `HOST_DAEMON_PROTOCOL_VERSION` stays **150**: nothing on the server↔daemon wire changed (translation and an item-shape change inside the `thread/delta` lane only). ## Regression oracle All turbo invocations with `--concurrency 4`; perf suite ignored (known-noisy, flagged on #2121). Old leg: an `origin/main` worktree at `f6fb434ab`. - **Conformance**: claude scripted suite + recorded conformance over all 14 recorded cells (incl. auth-failure, plan-mode, subagent, user-question) green. The claude `bridge→runtime.current.ndjson` lanes are re-recorded with this bridge (`pnpm rerecord --plan-with <main checkout> --provider claude-code`); the recordings themselves are untouched and the self-suite's row-count pins are **unchanged** (43/43). - **Parity (A2)**, `pnpm parity --old <origin/main worktree> --new . --provider claude-code`: **13 passed, 0 failed, 1 skipped** (process-scoped `model-list`). Event and row counts equal in every cell. `recordings/parity-allowlist.json` gains 27 entries naming `#2178`, three classes: - `presentation` on items — `/*/item/presentation`, events, 9 cells (approval-allow, approval-deny, compaction, plan-mode, steer, subagent, turn-tools, user-question, web-search; the other 4 cells have no items). - Read → `fileRead` — plan-mode events `/2`, `/4`, `/24`, `/25`; rows `/0/children/{0,5}/{toolName, toolArgs, output, activityIntents/0/command, activityIntents/0/name}` (the row projects from a fileRead item: no tool name or arguments, the file contents are not row data, the intent command is `Read <path>`). - Agent → `delegation` — subagent events `/6`, `/13`; rows `/0/children/0/toolName` ("delegation") and `/0/children/0/subagentType` (the delegation item has no such field; the type rides the presentation detail). Zero unlisted diffs, zero stale entries. codex **16/16** and acp-cursor **10/10** replay against main with **zero new diffs** (no entries added; nothing of theirs touched). The expected "suppressed low-value rows" and "planSteps" classes produce no parity diff: no recorded cell calls Monitor/TaskOutput/ScheduleWakeup/SendMessage or TodoWrite/Task*, and ToolSearch/AskUserQuestion were already hidden by name. - **Corpus (A4)**, 307 threads / 93,262 rows: **zero diffs**, claude-code and codex alike, no corpus allowlist entry needed. Persisted Claude events are `toolCall` items without presentation, and every thread-view change here applies only to the v3 kinds and to `presentation.suppress` — the legacy tables and reducer are untouched — so old rows project identically by construction. - **G11**: `provider/unhandled` flat on every claude cell (0→0; compaction and steer 2→2) and on the corpus; pinned per transcript fixture (23 across 12 fixtures, every one a CLI-injected string-content `user` message — compaction summaries, `<task-notification>` resumes — which visibility classifies `unknown` today; lowering that is a separate change). - **G1**: unchanged (no provider-id literal added or removed in core; the legacy Claude tool-name tables stay for persisted rows). Tests (forced): thread-view 385, claude-code 322, codex 193, provider-bridge-protocol 218, provider-parity 43; server 1832/1833 (the one failure is the known local umask case in `internal-skill-trees`, which passes in CI). Typecheck green for the claude plugin, thread-view, server and agent-runtime on the re-stacked base. ## Not done - **Live QA cells via bb-dev-app** (turn/steer/stop/approve/deny/question/subagent/resume/fork/plan with screenshots) — not run; the recorded cells, the transcript fixtures and parity were the oracle. - `planSteps` rows are not projected as timeline rows (status quo: TodoWrite/Task* rows were hidden); the banner reads them. The presentation-driven projection workstream owns the rows. - The delegation row loses the "(Explore)" sub-agent-type suffix for new Claude sub-agent rows until the projection reads the presentation detail (allowlisted; old rows unaffected). - Backgrounded `Agent` calls keep today's two-row structure (the delegation settles at the launch ack; the `local_agent` background task tracks the work). Folding the task into a single background delegation is a larger change to the runtime's open-work tracking and the background-commands card. - String-content `sdk/user` messages (task-notification resumes, compaction summaries) still surface as `provider/unhandled`; the fixtures show they are the whole G11 residue for Claude. A one-line visibility change would lower it but changes parity for the steer/compaction cells, so it is left for a deliberate follow-up. > AGENT GENERATED: by Claude Opus 5 --------- Co-authored-by: Claude <noreply@anthropic.com>
This was referenced Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
The provider-plugin migration abandons byte-equivalence with the old goldens on purpose, which removes the regression oracle. Before any provider code moves,
mainneeds machine checks on real data that every later layer can run: projected rows for the 307 production threads in the private corpus (A4), a timeline-build and event-size baseline, and the permission-decision matrix (A5/G12) pinned as a literal table before WS5 changes the unions. None of these existed.What changed
No production code changes. Tests, test helpers, one script, one turbo task, docs.
packages/test-helpers/src/provider-corpus.ts:corpusAvailable(),listCorpusThreads({ provider?, reasons? }),loadCorpusThread(id). ReadsBB_PROVIDER_CORPUS_DIR(manifest.json,threads/<provider>/<id>/{meta.json,events.ndjson}), validates rows at the boundary with zod (ids must be one safe path segment), resolves each thread through its manifest entry and fails unlessmeta.jsonand every event row agree on id, provider, reasons, and row count, and decodes event payloads with@bb/domainparseStoredThreadEvent+buildThreadEventRow— the same path as the server'sparseStoredEventRow..gitignoregets**/provider-corpus/**with re-includes for the two in-repo directories of that name.apps/server/test/provider-corpus/row-snapshots.test.ts+corpus-harness.ts. Each thread is inserted into in-memory SQLite with its original ids, sequences, and timestamps (rawINSERTso nothing is minted), then every timeline page is built viabuildThreadTimelineWithProfilewith the options the route uses — event budget fromdefaultFeatureFlags, inline-output limit fromDEFAULT_MAX_INLINE_OUTPUT_CHARS, display name and plan command from the real provider registry (createTestProviderRegistry()loads the first-party plugin declarations) throughresolveProviderPlanCommand, unhandled ops included, output truncation + preview — following the route's ownolderCursor. Two variants per thread:default(turn rows summarized) andnested(includeNestedRows, children materialized). Snapshots go to$BB_PROVIDER_CORPUS_DIR/snapshots/rows/<provider>/<threadId>.json, keys sorted. Nothing is blanked: no wall-clock value reaches the rows, and write mode proves it by projecting every thread twice and requiring byte equality. Compare mode fails on any diff not covered bysnapshots/allowlist.json(threadId|provider|"*"scope, JSON-pointer or*/**glob path,pr,reason), prints a unified diff for the first 3 differing threads plus a count, lists the entries it used, and fails on entries that cover nothing.timeline-perf.test.ts: 10 largest threads per provider, latest page and full page walk, 5 profiled builds each after a warm-up, stage p50s fromThreadTimelineBuildProfile, persisteddatabytes median/p95/total, rows produced. Written tosnapshots/perf-baseline.json; compare fails at baseline × 1.10 for build cost and × 1.15 for median event size. Deviation from the brief, with data: the gate uses a normalized cost — min build time ÷ min time of a fixed CPU workload that shares no code with the timeline (JSON codec + sort over a deterministic document, run once per sample right before the builds) — not raw p50/p95. Raw p50 of the same commit swung up to 30% between two back-to-back runs on this 16-core box at load ~6 (14 of 20 threads tripped a literal 1.10 gate on the very next run); each side's minimum discards its own contended samples, interleaving keeps both minima in one short window, and a workload outside the timeline path means a uniform regression still moves the ratio. Raw p50/p95 are still recorded and printed. Up to 3 attempts per thread (write mode keeps the median attempt; compare stops at the first pass), a 5 ms floor for tiny latest-page builds, and compare mode refuses a baseline written with different gate settings. The table header reports the load average and flags an oversubscribed machine.synthetic-thread.tsbuilds a 10,019-event thread (every item kind, deltas, background tasks, usage events) and walks all 12 pages. Gate: minimum of 5 walks under 1,500 ms (local minimum 150–170 ms; the ceiling is ~10× so a slow runner passes while a quadratic regression still fails).packages/agent-runtime/src/permission-matrix.test.ts: the runtime chokepointhandleRuntimeProviderRequestover permission policy (5 members of the discriminated union) × approval subject (5: command, file_change, permission_grant, plan, tool_use) ×approvalEnforcedBy(2) × deny availability (2) = 100 cells, each a literal;satisfies Record<CellKey, Outcome>plusSameUniontype guards against the domain unions (dropping a row or a union member failstsc, verified on the stack), plus runtime assertions that every local subject kind parses as a payload. The request goes through WS1a's real bridge-protocol adapter.apps/server/test/permissions/permission-matrix.test.ts:resolvePermissionEscalationover the 3 initiators and the 54-cell runtime-permission-policy shape cross product (5 accepted), with the reviewer vocabulary pinned to the policy union at the type level.scripts/provider-corpus/snapshot-rows.sh [write|compare],@bb/server#test:provider-corpus(uncacheable,passThroughEnvfor the two variables — strict turbo env mode strips them from the plaintesttask), and a "Provider corpus" section indocs/debugging-and-qa.md.Permission matrix (runtime)
Outcome:
forward= reachesonInteractiveRequest(user decides);auto-deny= runtime answers deny;encode-error= runtime wants to auto-deny butdenyis not inavailableDecisions, so the provider gets a JSON-RPC error. Subject kind (command, file_change, permission_grant, plan, tool_use) never changes the outcome, so the table is collapsed over it (each row below is 5 cells).Bridge-kit predicate:
shouldAutoDenyInteractiveRequest→ ask: false, deny: true, null: false.Server escalation by initiator: user → ask; agent → deny; system → deny. (On
mainthe function also took the thread and was measured identically for root, delegated-child, and fork threads; WS1a removed the unused argument.)Accepted runtime policy shapes (5 of 54): accept-edits/workspace/user/{ask,deny}, auto/workspace/automatic/{ask,deny}, full/full/–/–.
Observations (pinned, not fixed):
approvalPolicy/sandbox, Claude SDKpermissionMode) and enforced by the provider. So a runtime-enforced provider infullmode that did send an approval would prompt the user.encode-errorcells: on an agent- or system-initiated turn, a runtime-enforced provider whose approval omitsdenyreceives a JSON-RPC error instead of a decision. Codex forwards the provider's own decision list, so this is reachable in principle; no first-party bridge omits deny today.planapprovals are treated like any other subject by the runtime: on a system-initiated turn with a runtime-enforced provider they are auto-denied. Claude is provider-enforced and always forwardsExitPlanMode, so only codex plan approvals can hit this.How you verified
pnpm exec turbo run typecheck --filter=@bb/server --filter=@bb/test-helpers --filter=@bb/agent-runtime— 6 tasks successful.--concurrency 4):pnpm exec turbo run typecheck --filter=@bb/agent-runtime --filter=@bb/server --filter=@bb/test-helpersgreen;pnpm exec turbo run test --filter=@bb/agent-runtime --filter=@bb/test-helpers— 31 files, 434 tests passed (102 of them the matrix);pnpm exec turbo run test --filter=@bb/server— 197 files, 1,886 tests passed, the one failure again the local umask assertion. With the corpus set, the row snapshots minted onmaincompare byte-identical under WS1a's v3 assembler: 307/307, 0 diffs.mainbefore the stack, corpus absent:pnpm exec turbo run test --filter=@bb/agent-runtime --filter=@bb/test-helpers— 32 files, 504 tests passed (82 of them the matrix).pnpm exec turbo run test --filter=@bb/server— 196 files passed, 1 skipped (the row-snapshot suite), 1,884 tests passed; the single failure is the pre-existinginternal-skill-treesfile-mode assertion (this checkout's umask 0002 yields 0664 where the test expects 0644; it fails on cleanmainhere and passes in CI). The two corpus suites report as skipped; the synthetic benchmark runs (Synthetic 10019-event thread: 12 pages, 1165 rows projected, full walk p50 174 ms).scripts/provider-corpus/snapshot-rows.sh compare→ 2 files, 328 tests passed (307 row snapshots + 20 perf threads + 1 synthetic).parseStoredThreadEvent(426 MB ofdata)./variants/*/pages/*/rows/*/textfor that thread passes; a stale entry fails with "every snapshots/allowlist.json entry must cover at least one diff".TS2741 Property '"full/-|plan|provider|deny-unavailable"' is missing; dropping"provider"from the enforcer list →SameUnionbecomesfalseand thesatisfiesrejects the extra keys.Perf baseline (write mode, this commit, load 9.2/16 cores; ms; norm = min build ÷ min
json-sort-v1calibration)Part of the provider-plugin migration: the Step 0 baselines PR from the design spec's "Regression confidence" section. No tracking issue.