[stacked on #2189] Publish the parity replay harness through the plugin SDK testing kit

apps/server/src/services/skills/builtin-skills/bb-plugin-authoring/SKILL.md

@@ -1232,6 +1232,26 @@ assert what each row becomes
 import a private `@bb/*` package from a plugin: an installed plugin cannot
 resolve it.
 
+**Recorded replay.** The same kit ships the regression oracle the first-party
+bridges use. Record a real session: start the host daemon with
+`BB_PROVIDER_BRIDGE_RECORD_DIR=<dir>` in its environment, run a thread on
+your provider, and bb writes `<dir>/<providerId>/<threadId>/<direction>.ndjson`
+(a bridge that spawns a CLI also calls `experimental_recordProviderChildIo`
+right after `spawn()`). Commit the lanes under your plugin, then replay them in
+a test: `experimental_resolveProviderBridgeLaunch({ modulePath, pluginId })`
+builds the bridge process exactly as the runtime spawns it,
+`experimental_replayRecording` drives the recorded runtime lane into it and
+answers its requests with the recorded answers, and
+`experimental_compareParity` diffs the assembled events against the
+recording's own (`experimental_assembleRecordedEvents`); `experimental_checkRecordedCellReplay`
+adds the recorded-cell conformance verdicts. A bridge with a provider child
+passes a `ReplayProviderProfile` whose `env` (or `rewriteRuntimeLine`) points
+the child at the kit's replay script. When a deliberate bridge change alters
+the stream, `experimental_rerecordCurrentBridgeLane` writes the new
+expectation beside the recording (`bridge→runtime.current.ndjson`); the
+recording itself is never rewritten. See
+`examples/plugins/echo-provider/provider-bridge.parity.test.ts`.
+
 **Delivery.** On install/reload the server builds `dist/host.js` and records
 its digest. Thread commands for the provider carry `{pluginId, digest}` to the
 host daemon, which downloads the bytes from the server, verifies the digest

docs/api_to_audit.md

@@ -617,9 +617,23 @@ collector that feeds captured notifications through it
 `experimental_assembleCapturedThreadEvents`,
 `experimental_toConformanceMessages`); the in-process JSON-RPC harness
 (`experimental_captureBridgeJsonRpcOutput`,
-`experimental_createBridgeJsonRpcTestHarness`); and the calibration
+`experimental_createBridgeJsonRpcTestHarness`); the calibration
 normalizer (`experimental_normalizeCalibrationEvents`,
-`experimental_describeCalibrationEvents`).
+`experimental_describeCalibrationEvents`); and the recorded-replay harness —
+the regression oracle the first-party bridges use, keyed by the caller's
+provider id and bridge module rather than a list of bb's providers:
+`experimental_resolveProviderBridgeLaunch` (the bridge process as the
+runtime spawns it, through the bootstrap the kit ships beside its bundle),
+`experimental_replayRecording` (the recorded runtime lane in, the recorded
+provider lanes played by the kit's replay child, the bridge's output
+assembled), `experimental_assembleRecordedEvents`,
+`experimental_compareParity` / `experimental_normalizeParityEvents` /
+`experimental_normalizeParityRows` / `experimental_describeParityValue`,
+`experimental_replayRecordedCells` + `experimental_checkRecordedCellReplay`
+(the recorded-cell conformance verdicts), `experimental_rerecordCurrentBridgeLane`
+(the bridge's current output written beside a recording that is never
+rewritten), and the recording readers (`experimental_readBridgeRecording`,
+`experimental_listRecordedCells`, `experimental_withCurrentBridgeLane`).
 Framework-agnostic (the stdout capture patches `process.stdout.write`
 itself; nothing imports a test runner). Curated by hand, named exports only.
 The echo example and every first-party bridge suite import only this entry
@@ -641,9 +655,27 @@ protocol`'s `assembler`, `conformance`, and `testing` subpaths.
    (`turnId`, `itemId`, `id`, `parentToolCallId`) and drops
    `providerCheckpointId`. Confirm the defaults against a third-party
    bridge's goldens before fixing them.
-3. **Surface size.** 14 value exports plus 20 types. The JSON-RPC harness
+3. **Surface size.** 33 value exports plus 43 types. The JSON-RPC harness
    duplicates a little of the bridge kit's envelope parsing; fold or keep
    deliberately.
+4. **Replay profile shape.** `ReplayProviderProfile` is the three seams the
+   first-party bridges needed (`env`, `rewriteRuntimeLine`, `prepareState`)
+   and `ReplayDialect` is the two protocols the replay child speaks
+   (`json-rpc`, `claude-cli`). A third-party bridge whose CLI speaks neither
+   cannot replay its provider lanes. Decide whether the dialect set grows, or
+   whether the child becomes pluggable, before the profile is a promise.
+5. **Two shipped programs.** The kit's bundle spawns `provider-bridge-worker-entry.mjs`
+   and `replay-provider-child.mjs` from beside itself (`import.meta.url`),
+   so the published package carries both under `dist/`. Confirm the bundled
+   bootstrap tracks the daemon's (`apps/host-daemon/scripts/bundle-manifest.mjs`
+   builds the same entry) — a drift would make a replay differ from
+   production in argv or framing.
+6. **Workspace-path restoration.** `experimental_replayRecording` rewrites
+   this replay's temp workspace back to the recorded `cwd` in every line the
+   bridge emits, so a bridge that derives paths from `cwd` compares with the
+   recording. It is a textual substitution of a unique temp path; confirm
+   no bridge emits that path in a form the substitution misses (URL-encoded,
+   JSON-escaped backslashes on Windows).
 
 ## `app.experimental_useProviders` (`@get-bb/plugin-sdk/app`)
 

docs/provider-bridge-protocol.md

@@ -508,5 +508,23 @@ scenario set: `replayRecordedCells` replays a bridge's cells and
 `checkRecordedCellReplay` reports `recorded/<cell>/{replays,
 events-schema-valid, grammar, turn-lifecycle, not-empty}` per cell. Each
 first-party bridge has a `bridge.recorded-conformance.test.ts` beside its
-scripted suite, so conformance reflects the real dialect as well as the
-protocol.
+scripted suite (`replayFirstPartyRecordedCells`, which supplies the
+checkout's bridge module and the provider's replay profile), so conformance
+reflects the real dialect as well as the protocol.
+
+The harness is provider-agnostic and published. `parity.ts` knows nothing
+about which providers bb ships: a caller names the recording, the bridge
+process (`resolveProviderBridgeLaunch({ modulePath, pluginId })` builds one
+through the bootstrap, exactly as the runtime spawns it) and, for a bridge
+with a provider child, a `ReplayProviderProfile` that points the child at
+`replay-provider-child.mjs` (an `env` for a bridge that reads its CLI path
+from the environment, `rewriteRuntimeLine` for one that reads it from
+`thread/start`, `prepareState` for state the bridge reads from disk). The
+first-party profiles and module paths live in `testing/first-party-replay.ts`,
+private. `@get-bb/plugin-sdk/provider-bridge/testing` re-exports the core
+(`experimental_replayRecording`, `experimental_compareParity`,
+`experimental_assembleRecordedEvents`, `experimental_replayRecordedCells`,
+`experimental_rerecordCurrentBridgeLane`, the recording readers) and ships
+the bootstrap and the replay child beside its bundle, so a third-party
+plugin replays its own recordings with the same oracle
+(`examples/plugins/echo-provider/provider-bridge.parity.test.ts`).

examples/plugins/echo-provider/README.md

@@ -70,6 +70,14 @@ the same scripted turn:
      `item/tool/call` reply), runs the bridge's deltas through the **real**
      delta assembler the kit ships, and asserts every capability above on the
      assembled events.
+   - `provider-bridge.parity.test.ts` replays `recordings/echo-agent/turn-tools`
+     — a real recording bb made of this plugin's built artifact (record
+     mode, `BB_PROVIDER_BRIDGE_RECORD_DIR`) — through the bridge the way the
+     runtime spawns it, and diffs the assembled events against the
+     recording's own: zero diffs, every recorded-cell conformance rule green.
+     The same test re-records the bridge lane beside a copy of the recording,
+     the workflow a deliberate bridge change follows. This is the regression
+     oracle the first-party bridges use, reached through the public kit.
    - `public-sdk-only.test.ts` guards the rule.
 2. **The server**: `apps/server/test/providers/echo-provider-canary.test.ts`
    installs this plugin from its path, builds the real thread command, runs
@@ -111,3 +119,20 @@ sources, `bb plugin reload echo-provider`.
 ```
 pnpm exec turbo run test --filter=bb-plugin-echo-provider
 ```
+
+## Re-record
+
+The recording under `recordings/` is never rewritten. To capture a new one,
+start a dev bb with `BB_PROVIDER_BRIDGE_RECORD_DIR=<dir>` exported in the
+daemon's environment, install this plugin, spawn a thread on `echo-agent`,
+then package the thread's lanes:
+
+```
+printf 'echo-agent\tturn-tools\t<threadId>\t\n' > /tmp/cells.tsv
+node scripts/provider-recordings/package-cells.mjs --raw <dir> --cells /tmp/cells.tsv \
+  --out examples/plugins/echo-provider/recordings \
+  --versions '{"echo-agent":"bb-plugin-echo-provider 0.1.0"}'
+```
+
+The packager redacts the lanes (home paths, tokens, emails) and refuses to
+finish if a secret shape survives.

examples/plugins/echo-provider/provider-bridge.parity.test.ts

@@ -0,0 +1,234 @@
+/**
+ * The echo bridge's parity self-run: the recorded-replay oracle the
+ * first-party bridges regression-test with, reached through the published
+ * `@get-bb/plugin-sdk/provider-bridge/testing` kit alone.
+ *
+ * `recordings/echo-agent/turn-tools` is a real recording: bb's host daemon
+ * ran this plugin's built artifact with `BB_PROVIDER_BRIDGE_RECORD_DIR` set
+ * (docs/provider-bridge-protocol.md, "Record mode"), a thread was spawned on
+ * it, and the lanes were packaged and redacted with the recordings scripts.
+ * The runtime lane holds exactly what the runtime sent (`thread/start`,
+ * `turn/start`, the answer to the bridge's `item/tool/call`); the bridge
+ * lane holds exactly what the bridge emitted.
+ *
+ * The test spawns the bridge the way the runtime does (the bootstrap, the
+ * module, a plugin scope), drives the recorded runtime lane into it, answers
+ * its tool call with the recorded answer, assembles what it emits with the
+ * real delta assembler, and diffs that against the recording's own assembled
+ * events: zero diffs, zero grammar drops, and every recorded-cell conformance
+ * rule green. A bridge change that alters the stream for this session fails
+ * here first; `experimental_rerecordCurrentBridgeLane` then writes the new
+ * expectation beside the recording for the PR to explain.
+ *
+ * The echo bridge spawns no provider child, so no `ReplayProviderProfile` is
+ * needed: the recording's provider lanes are empty and the default profile
+ * applies. A bridge that drives a CLI supplies the env (or runtime-line
+ * rewrite) that points the CLI at the kit's replay child.
+ */
+import { cpSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+import { expect, it } from "vitest";
+import {
+  CURRENT_BRIDGE_LANE_FILE,
+  experimental_assembleRecordedEvents as assembleRecordedEvents,
+  experimental_checkRecordedCellReplay as checkRecordedCellReplay,
+  experimental_compareParity as compareParity,
+  experimental_createBridgeDeltaEventCollector as createBridgeDeltaEventCollector,
+  experimental_formatConformanceReport as formatConformanceReport,
+  experimental_listRecordedCells as listRecordedCells,
+  experimental_readBridgeRecording as readBridgeRecording,
+  experimental_replayRecording as replayRecording,
+  experimental_rerecordCurrentBridgeLane as rerecordCurrentBridgeLane,
+  experimental_resolveProviderBridgeLaunch as resolveProviderBridgeLaunch,
+  experimental_withCurrentBridgeLane as withCurrentBridgeLane,
+} from "@get-bb/plugin-sdk/provider-bridge/testing";
+import type {
+  CreateParityAssembler,
+  RecordedCell,
+} from "@get-bb/plugin-sdk/provider-bridge/testing";
+
+const packageRoot = dirname(fileURLToPath(import.meta.url));
+const RECORDINGS_ROOT = join(packageRoot, "recordings");
+const BRIDGE_MODULE = join(packageRoot, "src", "provider-bridge.ts");
+const PLUGIN_ID = "echo-provider";
+
+/** Every item kind the scripted turn emits; the recording must cover them all. */
+const SCRIPTED_ITEM_KINDS = [
+  "agentMessage",
+  "commandExecution",
+  "delegation",
+  "extension",
+  "fileRead",
+  "planSteps",
+  "search",
+  "toolCall",
+];
+
+/** The runtime adapter's exact delta→event translation, one event per delta. */
+const createAssembler: CreateParityAssembler = (providerId) => {
+  const collector = createBridgeDeltaEventCollector(providerId);
+  return { assembleMessage: (message) => collector.assembleMessage(message) };
+};
+
+const cells = listRecordedCells(RECORDINGS_ROOT);
+
+function cellKey(cell: RecordedCell): string {
+  return `${cell.provider}/${cell.cell}`;
+}
+
+it("ships a recorded cell for the echo provider", () => {
+  expect(cells.map(cellKey)).toEqual(["echo-agent/turn-tools"]);
+  const manifest = readBridgeRecording(cells[0]!.dir).manifest;
+  expect(manifest).toMatchObject({
+    provider: "echo-agent",
+    cell: "turn-tools",
+    scope: "thread",
+  });
+});
+
+it.each(cells.map((cell) => [cellKey(cell), cell] as const))(
+  "%s replays through the current bridge with zero diffs",
+  async (_key, cell) => {
+    // The recording's own view: what its bridge lane assembles to, no bridge
+    // in the loop. The current lane (a deliberate re-recording) wins when
+    // one exists beside the recorded lane.
+    const recorded = assembleRecordedEvents(
+      withCurrentBridgeLane(readBridgeRecording(cell.dir)),
+      createAssembler,
+      cell.provider,
+    );
+    expect(recorded.invalidDeltas).toEqual([]);
+    expect(recorded.grammarViolations).toEqual([]);
+    const recordedKinds = new Set(
+      recorded.events.flatMap((event) =>
+        event.type === "item/completed" ? [event.item.type] : [],
+      ),
+    );
+    expect([...recordedKinds].sort()).toEqual(SCRIPTED_ITEM_KINDS);
+
+    const run = await replayRecording({
+      recordingDir: cell.dir,
+      providerId: cell.provider,
+      bridge: resolveProviderBridgeLaunch({
+        modulePath: BRIDGE_MODULE,
+        pluginId: PLUGIN_ID,
+      }),
+      createAssembler,
+      planFromCurrentLane: true,
+      // Generous: the bridge boots through a TypeScript loader, and a busy
+      // CI runner can take a while to spawn it.
+      timeoutMs: 60_000,
+      onStderr: (text) => process.stderr.write(`[echo bridge] ${text}`),
+    });
+    expect(run.stalls).toEqual([]);
+    expect(run.exitCode).toBe(0);
+
+    const comparison = compareParity(
+      {
+        events: recorded.events,
+        rows: [],
+        grammarViolations: recorded.grammarViolations,
+      },
+      { events: run.events, rows: [], grammarViolations: run.grammarViolations },
+      [],
+      { provider: cell.provider, cell: cell.cell },
+    );
+    expect({ events: comparison.events, grammar: comparison.grammar }).toEqual({
+      events: { onlyInOld: [], onlyInNew: [] },
+      grammar: { onlyInOld: [], onlyInNew: [] },
+    });
+    expect(comparison.passed).toBe(true);
+    expect(run.events.length).toBe(recorded.events.length);
+
+    const results = checkRecordedCellReplay({
+      provider: cell.provider,
+      cell: cell.cell,
+      events: run.events,
+      recordedEvents: recorded.events,
+      stalls: run.stalls,
+    });
+    console.info(
+      `echo recorded conformance:\n${formatConformanceReport({
+        results,
+        passed: results.every((result) => result.status === "pass"),
+      })}`,
+    );
+    expect(
+      results
+        .filter((result) => result.status !== "pass")
+        .map((result) => `${result.id}: ${result.detail}`),
+    ).toEqual([]);
+  },
+  120_000,
+);
+
+it("re-records the bridge lane beside a copy of the recording and replays from it", async () => {
+  // The workflow a bridge change follows: write the bridge's current output
+  // next to the recording (the recording itself is never rewritten), then the
+  // self-run compares against that lane. On an unchanged bridge the new lane
+  // assembles to the recorded events, keeps the recording's workspace path,
+  // and names the recorded request id, so the recorded runtime answer still
+  // matches.
+  const cell = cells[0]!;
+  const copy = mkdtempSync(join(tmpdir(), "bb-echo-rerecord-"));
+  try {
+    cpSync(cell.dir, copy, { recursive: true });
+    const result = await rerecordCurrentBridgeLane({
+      recordingDir: copy,
+      providerId: cell.provider,
+      bridge: resolveProviderBridgeLaunch({
+        modulePath: BRIDGE_MODULE,
+        pluginId: PLUGIN_ID,
+      }),
+      createAssembler,
+      timeoutMs: 60_000,
+    });
+    expect(result.stalls).toEqual([]);
+    expect(result.file).toBe(join(copy, CURRENT_BRIDGE_LANE_FILE));
+
+    const lane = readFileSync(join(copy, CURRENT_BRIDGE_LANE_FILE), "utf8");
+    const recordedCwd = (
+      JSON.parse(
+        readBridgeRecording(cell.dir).entries.find(
+          (entry) => entry.dir === "runtime→bridge",
+        )!.line,
+      ) as { params: { cwd: string } }
+    ).params.cwd;
+    expect(lane).toContain(recordedCwd);
+    expect(lane).not.toContain("bb-parity-ws-");
+    const toolCallIds = lane
+      .split("\n")
+      .filter((raw) => raw.length > 0)
+      .map((raw) => JSON.parse((JSON.parse(raw) as { line: string }).line) as {
+        id?: string;
+        method?: string;
+      })
+      .filter((message) => message.method === "item/tool/call")
+      .map((message) => message.id);
+    expect(toolCallIds).toEqual(["echo-req-1"]);
+
+    const recorded = assembleRecordedEvents(
+      readBridgeRecording(cell.dir),
+      createAssembler,
+      cell.provider,
+    );
+    const current = assembleRecordedEvents(
+      withCurrentBridgeLane(readBridgeRecording(copy)),
+      createAssembler,
+      cell.provider,
+    );
+    expect(current.invalidDeltas).toEqual([]);
+    const comparison = compareParity(
+      { events: recorded.events, rows: [] },
+      { events: current.events, rows: [] },
+      [],
+      { provider: cell.provider, cell: cell.cell },
+    );
+    expect(comparison.events).toEqual({ onlyInOld: [], onlyInNew: [] });
+    expect(current.events.length).toBe(recorded.events.length);
+  } finally {
+    rmSync(copy, { recursive: true, force: true });
+  }
+}, 120_000);