Experimental: preserve parent prompt-cache prefixes for Desktop side chats

docs-site/src/content/docs/reference/configuration/providers.md

@@ -162,6 +162,7 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity
 | `modelVercelGatewayRouting?` | `Record<string, VercelGatewayRouting>` | Exact model-id overrides that replace the provider-wide Vercel AI Gateway preference. |
 | `authMode?` | `"key" \| "forward" \| "oauth" \| "local"` | Authentication mode (default `key`). OAuth/subscription credentials are stored outside `config.json`; `local` is limited to providers whose registry entry permits it. |
 | `codexAccountMode?` | `"pool" \| "direct"` | Canonical `openai` only; defaults to Pool. Direct bypasses pool state. |
+| `experimentalCodexSideChatCache?` | `boolean` | Experimental, default `false`. Canonical `openai` only. Allows eligible Desktop side chats to reuse a completed parent request’s prompt-cache identity. See [side-chat cache reuse](#experimental-desktop-side-chat-cache-reuse). |
 | `refreshPolicy?` | `"proactive" \| "lazy-only" \| "disabled"` | Override this OAuth provider's Token Guardian policy. |
 | `reasoningEfforts?` | `string[]` | Provider-wide Codex reasoning labels to advertise and send. For `google`-adapter providers, a configured ladder also asserts `thinkingLevel` capability: direct and Vertex non-image requests send the selected effort as `generationConfig.thinkingConfig.thinkingLevel`, while Cloud Code Assist uses its envelope-specific path. |
 | `modelReasoningEfforts?` | `Record<string, string[]>` | Per-model labels. An empty list hides effort control. As with `reasoningEfforts`, each configured `google`-adapter ladder asserts `thinkingLevel` capability; direct and Vertex non-image requests use the flat Gemini path, while Cloud Code Assist sends it under its request envelope. |
@@ -1003,3 +1004,101 @@ or expiry does not extend the history-recovery contract.
 Sender and recipient on routed Responses are context for the receiving model, not a new
 machine-readable routing protocol. Tool routing continues to use the existing collaboration
 contracts.
+
+
+## Experimental Desktop side-chat cache reuse
+
+Set `experimentalCodexSideChatCache: true` on the existing `providers.openai`
+configuration row, then restart the proxy. The default is disabled. Set it to
+`false` and restart to disable it and discard the process-local cache metadata.
+This option applies only to the canonical ChatGPT forward Responses provider.
+
+With this option enabled, the proxy observes completed streamed requests and
+keeps bounded fingerprints for up to 64 tasks, with a ten-minute lifetime and a
+2,048-input-item limit. It uses explicit Desktop fork metadata to match a side
+chat to its parent. The selected credential, account, model, settings, tools,
+and inherited prompt prefix must be compatible before the proxy reuses the
+parent's prompt-cache key and provider session identity. Child task and turn
+identifiers remain distinct. Failed or unfinished requests do not seed reuse.
+The snapshot fingerprints the parent request input, not its response output. Only
+the common observed prefix is verified and counted as matched. A side chat can
+carry the parent’s last answer or additional inherited items after that prefix;
+those items remain its own unchanged suffix, rather than becoming verified parent
+input or being replaced by stored parent content.
+Recognized stream obfuscation and reasoning-summary delivery options are excluded
+from cache identity checks, while each request retains its own options on the wire.
+Unknown or malformed stream options still require an exact match.
+
+The proxy recognizes exact Desktop side-conversation rule and boundary text.
+It moves recognized rules to a developer message at the side boundary, or adds
+a developer copy of the recognized boundary when no separate rule block exists.
+It also moves a small allowlist of context-dependent `functions.exec` method
+references to a final developer message containing that request's own methods.
+Executable tool schemas remain intact. An explicitly bounded inherited history
+may reuse its proven prefix before a differing reasoning item; the child's
+reasoning and subsequent messages remain unchanged.
+
+These transformations depend on the Desktop prompt format and need validation
+when that format changes. Unknown instruction differences, incompatible inputs,
+missing parents, continuations, and compaction requests skip parent reuse.
+Nested side chats can also skip when inherited transformations no longer match.
+Account switching or credential refresh can prevent a match. Upstream cache
+retention and hits are opportunistic; enabling this option does not guarantee a
+hit. Existing provider debug diagnostics report reason codes, opaque task tags,
+and token counts without recording prompt text or credentials.
+
+
+### Monitor side-chat cache reuse
+
+When the experimental setting is enabled, eligible adapter preparations include `sideChatCache`
+in the existing local `usage.jsonl` request and attempt records. No additional database or telemetry
+service is created. Missing metadata can mean an uninstrumented version, a disabled feature, or an
+adapter path that does not prepare side-chat reuse; it is not a measured miss.
+
+The fixed `reason` describes the reuse decision. `phase` distinguishes parent observations, side
+requests with no binding yet (`unbound-side`), and requests with an existing binding (`bound-side`).
+An unbound request is not necessarily the first-ever side request: restarts, expiry, eviction, and
+failed requests can remove or prevent a binding. `matchedItems` counts the verified inherited prefix.
+
+`snapshotOutcome` records whether a completed response stored the snapshot, or whether it expired,
+was superseded by a newer completion, or belonged to a disabled cache. `not-observed` means no accepted
+successful completion was recorded; it must not be interpreted as a stored parent. A reused prefix
+and a stored snapshot do not prove that the upstream returned cached tokens.
+
+`prepareMs` measures preparation including instrumentation. `normalizeMs` covers execution-reference
+normalization, `hashMs` accumulates fingerprint work, and `matchMs` covers candidate matching and
+prefix rewriting. **Hash time overlaps match time**, so do not add the phases. `completionMs` measures
+snapshot publication and pruning. Each attempt holds its last preparation and observed completion;
+multiple sends or rebuilds are not a cumulative timing trace.
+
+Retention fields count unique retained snapshots and bindings, plus estimated retained UTF-8 payload
+bytes. They exclude JavaScript object overhead and in-flight requests, and are not process heap usage.
+Expiry and eviction counts describe map entries removed during the recorded operations. Snapshots
+come from those operations, not a live memory query. `observedAt` timestamps the measurement;
+retention reports use it rather than request start time when completions arrive out of order. The optional child `threadIdHash` uses the same
+SHA-256 prefix convention as log conversation IDs and permits exact child correlation without storing
+a raw thread ID. No prompts, tool descriptions, credentials, or raw account identifiers are added.
+
+Summarize the newest usage rows from a source checkout:
+
+```bash
+bun scripts/side-chat-cache-report.ts 1000
+```
+
+An optional second argument selects an exact request ID within the bounded window. `OPENCODEX_HOME`
+selects another installation. The report counts attempts once, separates reported cache reads from
+unknown/estimated usage, and groups results by parent/unbound/bound phase. Cached-token ratios and
+first-output latency are observations; they do not establish which feature caused a cache hit.
+
+Run isolated synthetic control/treatment measurements without model API calls:
+
+```bash
+bun scripts/side-chat-cache-eval.ts .tmp/side-cache-eval 20 4
+```
+
+The harness compares the existing setting off/on with concurrent HTTP and WebSocket clients, 1 KiB,
+64 KiB, and 1 MiB inherited text, plus direct large-history and reordered-tool-catalog workloads.
+It writes `report.json` and `samples.jsonl`, recording the source commit, dirty status, full Bun build
+identity, and observed upstream transports. Fixture WebSocket availability does not imply its use:
+runtime gates can select HTTP fallback. Synthetic usage counters are fixtures, never measured cache
+savings. Run on the target operating system and validate actual Desktop behavior with ordinary usage.

scripts/side-chat-cache-eval.ts

@@ -0,0 +1,94 @@
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { distribution, summarizeSideChatCache } from "./side-chat-cache-report";
+
+const [mode, ...args] = process.argv.slice(2);
+if (mode === "--cell") {
+  const [enabledArg, nativeArg, turnsArg, concurrencyArg, bytesArg] = args;
+  const [{ startCacheProxy }, { SIDE_CHAT_BOUNDARY }, { readRecentUsageEntries }] = await Promise.all([
+    import("../tests/helpers/side-chat-cache-proxy"), import("../src/codex/side-chat-cache"), import("../src/usage/log")]);
+  const fixture = await startCacheProxy(nativeArg === "true", enabledArg === "true");
+  const message = (role: string, text: string) => ({ type: "message", role, content: [{ type: "input_text", text }] });
+  const history = [message("developer", "Synthetic rules"), message("user", "x".repeat(Number(bytesArg)))];
+  const body = (thread: string, input: unknown[], parent?: string) => ({ model: "gpt-5.6-luna", instructions: "Synthetic", input, stream: true, store: false,
+    prompt_cache_key: thread, client_metadata: { thread_id: thread, session_id: thread, ...(parent ? { forked_from_thread_id: parent } : {}) } });
+  const samples: Array<{ transport: string; phase: string; wallMs: number }> = [];
+  try {
+    const parent = await fixture.http(body("parent", history), "parent");
+    for (const transport of ["http", "websocket"]) {
+      await Promise.all(Array.from({ length: Number(concurrencyArg) }, async (_, i) => {
+        const thread = `${transport}-${i}`;
+        const ws = transport === "websocket" ? fixture.websocket(thread, "parent") : undefined;
+        let input = [...history, ...parent.output, message("user", SIDE_CHAT_BOUNDARY), message("user", "Side question")];
+        try {
+          for (let turn = 0; turn < Number(turnsArg); turn++) {
+            const request = body(thread, input, "parent");
+            const started = performance.now();
+            const response = ws ? await ws.turn(request) : await fixture.http(request, thread, false, "parent");
+            samples.push({ transport, phase: turn ? "follow-up" : "first-observed-side", wallMs: performance.now() - started });
+            input = [...input, ...response.output, message("user", "Next")];
+          }
+        } finally { ws?.close(); }
+      }));
+    }
+    console.log(JSON.stringify({ samples, observedUpstreamTransports: [...new Set(fixture.captured.map(row => row.transport))].sort(),
+      measurements: summarizeSideChatCache(readRecentUsageEntries(10_000, fixture.home)) }));
+  } finally { await fixture.stop(); }
+} else {
+  const [turnsArg = "20", concurrencyArg = "4"] = args;
+  const turns = Number(turnsArg), concurrency = Number(concurrencyArg);
+  if (!mode || !Number.isSafeInteger(turns) || turns < 2 || turns > 200 || !Number.isSafeInteger(concurrency) || concurrency < 1 || concurrency > 16) {
+    throw new Error("Usage: bun scripts/side-chat-cache-eval.ts <outDir> [2..200 turns] [1..16 clients]");
+  }
+  mkdirSync(mode, { recursive: true, mode: 0o700 });
+  const home = mkdtempSync(join(tmpdir(), "side-eval-"));
+  mkdirSync(join(home, "codex"));
+  const cells = [];
+  const direct = [];
+  try {
+    const { SideChatCache, SIDE_CHAT_BOUNDARY } = await import("../src/codex/side-chat-cache");
+    for (const [items, tools] of [[4, 16], [128, 128], [1024, 256]]) {
+      const cache = new SideChatCache();
+      const catalog = { type: "additional_tools", role: "developer", tools: Array.from({ length: tools }, (_, i) => ({ type: "function", name: `tool_${i}`, description: "Synthetic contract", parameters: { type: "object", properties: {} } })) };
+      const history = Array.from({ length: items }, (_, i) => ({ role: i ? "user" : "developer", content: `Synthetic item ${i}` }));
+      const headers = (thread: string) => ({ authorization: "Bearer synthetic", "chatgpt-account-id": "synthetic-account", "thread-id": thread, "session-id": thread });
+      const base = { model: "gpt-5.6-luna", instructions: "Synthetic", stream: true, store: false, prompt_cache_key: "parent", input: [catalog, ...history] };
+      cache.prepare(base, headers("parent")).complete();
+      const samples = [];
+      for (let i = 0; i < turns; i++) {
+        const request = { ...base, prompt_cache_key: "child", client_metadata: { forked_from_thread_id: "parent" },
+          input: [{ ...catalog, tools: [...catalog.tools].reverse() }, ...history, { role: "user", content: SIDE_CHAT_BOUNDARY }, { role: "user", content: "Side question" }] };
+        const decision = cache.prepare(request, headers("child"));
+        decision.complete();
+        const { threadIdHash: _thread, ...sample } = decision.metrics;
+        samples.push(sample);
+      }
+      direct.push({ inputItems: items, catalogTools: tools, samples });
+    }
+    for (const bytes of [1024, 64 * 1024, 1024 * 1024]) {
+      for (const native of [false, true]) {
+        for (const enabled of [false, true]) {
+          const child = Bun.spawn([process.execPath, import.meta.path, "--cell", String(enabled), String(native), String(turns), String(concurrency), String(bytes)],
+            { env: { ...process.env, HOME: home, USERPROFILE: home, OPENCODEX_HOME: home, CODEX_HOME: join(home, "codex") }, stdout: "pipe", stderr: "pipe" });
+          const timeout = setTimeout(() => child.kill(), 120_000);
+          try {
+            const [stdout, stderr, code] = await Promise.all([new Response(child.stdout).text(), new Response(child.stderr).text(), child.exited]);
+            if (code !== 0) throw new Error(`Synthetic side-cache cell failed (${code}): ${stderr.slice(-1000)}`);
+            const result = JSON.parse(stdout.trim().split("\n").at(-1)!);
+            cells.push({ enabled, upstreamWebSocketAvailable: native, inputTextBytes: bytes, clients: concurrency, ...result });
+          } finally { clearTimeout(timeout); if (child.exitCode === null) { child.kill(); await child.exited; } }
+        }
+      }
+    }
+    const head = Bun.spawnSync(["git", "rev-parse", "HEAD"], { cwd: join(import.meta.dir, "..") });
+    const status = Bun.spawnSync(["git", "status", "--porcelain"], { cwd: join(import.meta.dir, "..") });
+    if (head.exitCode || status.exitCode) throw new Error("Cannot identify benchmark checkout");
+    writeFileSync(join(mode, "samples.jsonl"), [...cells.flatMap((cell, index) => cell.samples.map((sample: unknown) => JSON.stringify({ cell: index, sample }))), ...direct.flatMap((cell, index) => cell.samples.map(sample => JSON.stringify({ direct: index, sample })))].join("\n") + "\n");
+    writeFileSync(join(mode, "report.json"), JSON.stringify({ schemaVersion: 1, synthetic: true, platform: process.platform, arch: process.arch,
+      bunVersionWithSha: Bun.version_with_sha, commit: head.stdout.toString().trim(), dirty: status.stdout.length > 0,
+      direct: direct.map(({ samples, ...cell }) => ({ ...cell, prepareMs: distribution(samples.map(row => row.prepareMs)), last: samples.at(-1) })),
+      cells: cells.map(({ samples, ...cell }) => ({ ...cell, wallMs: distribution(samples.map((row: { wallMs: number }) => row.wallMs)) })) }, null, 2) + "\n");
+    console.log("Synthetic benchmark complete: report.json and samples.jsonl");
+  } finally { rmSync(home, { recursive: true, force: true }); }
+}

scripts/side-chat-cache-report.ts

@@ -0,0 +1,56 @@
+import { readRecentUsageEntries, type PersistedUsageEntry } from "../src/usage/log";
+import { normalizeSideChatCacheMetrics } from "../src/usage/side-chat-cache";
+
+export function distribution(values: number[]) {
+  const sorted = [...values].sort((a, b) => a - b);
+  const at = (p: number) => sorted.length ? sorted[Math.ceil(p * sorted.length) - 1] : null;
+  return { samples: sorted.length, p50: at(0.5), p95: at(0.95), p99: at(0.99), max: sorted.at(-1) ?? null };
+}
+
+export function summarizeSideChatCache(entries: PersistedUsageEntry[]) {
+  const reasons: Record<string, number> = {}, phases: Record<string, number> = {}, snapshotOutcomes: Record<string, number> = {};
+  const timings: Record<string, number[]> = { prepareMs: [], completionMs: [], normalizeMs: [], hashMs: [], matchMs: [], requestOrAttemptMs: [], firstOutputMs: [] };
+  const cache = { hit: 0, miss: 0, unknown: 0, invalid: 0, noInput: 0, inputTokens: 0, cachedInputTokens: 0 };
+  const byPhase: Record<string, { samples: number; hits: number; misses: number; unknown: number }> = {};
+  let samples = 0, expiredEntries = 0, evictedEntries = 0;
+  let latestRetention: { observedAt: number; requestTimestamp: number; retainedSnapshots: number; retainedBindings: number; estimatedRetainedBytes: number } | null = null;
+  for (const entry of entries) {
+    for (const row of entry.attempts?.length ? entry.attempts : [entry]) {
+      const metrics = normalizeSideChatCacheMetrics(row.sideChatCache);
+      if (!metrics) continue;
+      samples++;
+      reasons[metrics.reason] = (reasons[metrics.reason] ?? 0) + 1;
+      phases[metrics.phase] = (phases[metrics.phase] ?? 0) + 1;
+      snapshotOutcomes[metrics.snapshotOutcome] = (snapshotOutcomes[metrics.snapshotOutcome] ?? 0) + 1;
+      const phase = byPhase[metrics.phase] ??= { samples: 0, hits: 0, misses: 0, unknown: 0 };
+      phase.samples++;
+      timings.prepareMs.push(metrics.prepareMs);
+      for (const key of ["completionMs", "normalizeMs", "hashMs", "matchMs"] as const) if (metrics[key] !== undefined) timings[key].push(metrics[key]);
+      for (const [key, value] of [["requestOrAttemptMs", row.durationMs], ["firstOutputMs", row.firstOutputMs]] as const) {
+        if (typeof value === "number" && Number.isFinite(value) && value >= 0) timings[key].push(value);
+      }
+      expiredEntries += metrics.expiredEntries; evictedEntries += metrics.evictedEntries;
+      if (metrics.observedAt !== undefined && (!latestRetention || metrics.observedAt >= latestRetention.observedAt)) latestRetention = { observedAt: metrics.observedAt, requestTimestamp: entry.timestamp,
+        retainedSnapshots: metrics.retainedSnapshots, retainedBindings: metrics.retainedBindings, estimatedRetainedBytes: metrics.estimatedRetainedBytes };
+      const input = row.usage?.inputTokens, cached = row.usage?.cachedInputTokens;
+      if (row.usageStatus !== "reported" || input === undefined || cached === undefined) { cache.unknown++; phase.unknown++; }
+      else if (!Number.isSafeInteger(input) || !Number.isSafeInteger(cached) || input < 0 || cached < 0 || cached > input) { cache.invalid++; phase.unknown++; }
+      else if (input === 0) { cache.noInput++; phase.unknown++; }
+      else {
+        cache[cached > 0 ? "hit" : "miss"]++; phase[cached > 0 ? "hits" : "misses"]++;
+        cache.inputTokens += input; cache.cachedInputTokens += cached;
+      }
+    }
+  }
+  return { rowsRead: entries.length, samples, reasons, phases, snapshotOutcomes, byPhase, expiredEntries, evictedEntries, latestRetention,
+    timingsMs: Object.fromEntries(Object.entries(timings).map(([key, values]) => [key, distribution(values)])),
+    cache: { ...cache, cachedInputRatio: cache.inputTokens ? cache.cachedInputTokens / cache.inputTokens : null } };
+}
+
+if (import.meta.main) {
+  const [limitArg = "1000", requestId] = process.argv.slice(2);
+  const limit = Number(limitArg);
+  if (!Number.isSafeInteger(limit) || limit < 1 || limit > 10_000) throw new Error("Usage: bun scripts/side-chat-cache-report.ts [1..10000 recent rows] [request-id]");
+  const rows = readRecentUsageEntries(limit);
+  console.log(JSON.stringify(summarizeSideChatCache(requestId ? rows.filter(row => row.requestId === requestId) : rows), null, 2));
+}