diff --git a/CHANGELOG.md b/CHANGELOG.md index c56f87ad..3a952829 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## [Unreleased] +### Added + +- **`CACHE_FIX_COALESCE_SIDECAR=1` serves CC's duplicated session-start sidecar send from one upstream call instead of two.** CC issues that request twice, 6–25 ms apart; both are answered and both are charged (measured: 144 pairs, 114 streaks, 55 double-billed sends, 48,203 input-side tokens). Dropping the second is unavailable — two client requests are in flight and each is owed a response — so the second caller attaches to the first call's response stream and receives it in full, mid-stream attaches included via replay. Four conditions gate it, and the mid-session duplicate class (where a second send is a legitimate retry) fails on message count alone. Default-off. The coalescing key covers the caller's credential headers as well as the forwarded bytes, so on a shared proxy two users' byte-identical sidecars never share a call — see [Duplicate sidecar coalescing](README.md#duplicate-sidecar-coalescing-proxy-mode-opt-in) for the shared-proxy posture, including the stated behaviour when a request carries no credential at all. + ### Changed - **`CACHE_FIX_REQUIRE_HOP=1` now covers the relayed `/v1/messages` route, not just the two `CONNECT` paths.** With the variable set and no chain hop reachable, that route previously dialled `api.anthropic.com` directly — carrying the caller's API key past the boundary the variable exists to enforce, while the `CONNECT` paths correctly refused. It now answers `502` instead, matching them. This is a new user-visible outcome on the primary route: an operator who sets the variable, configures no fallbacks and has no reachable proxy will see requests refused where they previously succeeded, which is what the flag asks for. Unset (the default) nothing changes. Hosts exempted by `NO_PROXY` stay exempt — that is an operator saying "this one is direct on purpose". Three egress sites still do not consult the variable: `storageAgent()`, the update-channel probe, and `fallbackToOrigin()`. The first two issue our own requests and carry no client headers. The third forwards the client's headers verbatim to `downloads.claude.ai` on the opt-in download-rewrite path — it is enumerated here rather than claimed harmless. diff --git a/README.md b/README.md index 5daadb09..5034d8ea 100644 --- a/README.md +++ b/README.md @@ -455,6 +455,7 @@ All proxy settings are controlled via environment variables. Set them before sta | `CACHE_FIX_DEBUG` | `0` | Enable debug logging | | `CACHE_FIX_HOT_RELOAD` | unset | Set to `on` to enable in-process extension hot-reload. Off by default as of v4.0.0 — see [Upgrading from v3.x](#upgrading-from-v3x) for details and the supervisor restart flow. | | `CACHE_FIX_READ_DEDUPE` | unset | Set to `1` to dedupe repeat `Read` tool results that re-appear unchanged across turns. Keeps the first occurrence intact; replaces later byte-identical ones (keyed on `file_path` + content + `offset` + `limit`) with a stable pointer line. Default-off; opt in per session to validate before broader rollout. See [extension impact guide](docs/extension-impact-guide.md). | +| `CACHE_FIX_COALESCE_SIDECAR` | unset | Set to `1` to serve CC's duplicated session-start sidecar send from one upstream call instead of two. Default-off. See [Duplicate sidecar coalescing](#duplicate-sidecar-coalescing-proxy-mode-opt-in). | | `CACHE_FIX_ADVISOR_PLAN` | unset | Plan override for `tools/tier-advisor.mjs` — one of `max-5x`, `max-20x`, `pro`. Bypasses heuristic plan detection. See [Tier advisor](docs/tier-advisor.md). | | `CACHE_FIX_ADVISOR_UPGRADE_THRESHOLD` | `80` | Projected-Q7d percent that triggers an `UPGRADE` recommendation from tier-advisor. | | `CACHE_FIX_ADVISOR_DOWNGRADE_THRESHOLD` | `20` | Projected-Q7d percent that triggers a `DOWNGRADE` recommendation from tier-advisor (paired with the `DOWNGRADE_WEEKS` consecutive-weeks gate). | @@ -1174,6 +1175,37 @@ export CACHE_FIX_NORMALIZE_CC_VERSION=pin:2.1.185 # operator-supplied literal The extension runs at order 90, before `fingerprint-strip` at order 100. After normalization the `cc_version` has at most 3 segments, so `fingerprint-strip`'s `dotParts.length < 4` guard makes it a no-op — the two cooperate cleanly with no other ordering hazards. Field-boundary anchored regex `(^|[;\s:])cc_version=([^;\s]+)` so a `cc_version=` substring embedded in another field's value cannot be accidentally rewritten. Atomic fail-open: planned rewrites stage in a local array and apply only after the scan completes; any error during the scan leaves the body byte-intact. +## Duplicate sidecar coalescing (proxy mode, opt-in) + +At session start Claude Code issues one small "sidecar" request **twice**, 6–25 ms apart. Both sends reach the API, both get distinct upstream request-ids, and both produce a completed usage-log record — so both are charged. Measured over one corpus: 144 pairs, 114 streaks, 55 double-billed sends, 48,203 input-side tokens. + +Dropping the second send is not available: two client requests are in flight and each is owed a response. The only safe shape is **one upstream call serving both callers**, which is what this does — the second caller attaches to the first call's response stream and receives it in full (including mid-stream, via replay of what the leader has already written). + +Opt-in via env var; default-off: + +```bash +export CACHE_FIX_COALESCE_SIDECAR=1 +``` + +**Four conditions, all required.** The mid-session duplicate class — where a second identical send is a legitimate retry and answering it from a sibling would leave a real request unanswered — fails on `nMsg` alone, which is the discriminator: + +| # | Condition | +|---|-----------| +| 1 | Exactly one message in `messages[]` | +| 2 | No `tools[]` (an empty array counts as none) | +| 3 | Byte-identical **forwarded** bodies (after every extension has run) **from the same caller** | +| 4 | The second send arrives < 50 ms later, with the first still in flight | + +**Why substituting one answer for the other is fidelity-safe**, which is the objection that kept this parked: a request carrying no conversation history and no tools produces output that can enter no cached prefix, and CC issues the second send before it could have observed the first — so it already treats the two as interchangeable. + +### Shared-proxy posture + +Condition 3's "from the same caller" half is what makes this safe to run on a proxy more than one person's credentials pass through. The sidecar's shape is fixed, so two different users' sends are **byte-identical**; a body-only coalescing key would let one user's in-flight call answer the other's request — the leader's account billed for both, a leader's `401` propagated to a follower holding valid credentials, and the leader's plan-tier / org context in bytes the follower reads. + +The coalescing key is therefore a digest of the request's **credential headers** (`authorization`, `x-api-key`, `proxy-authorization`, `cookie`) **and** the forwarded bytes. Two requests coalesce only when both halves match, so a pair from different callers never meets. The credential is hashed, never stored or logged — only the resulting key's first 16 characters reach the debug log. + +One case stated rather than left to be discovered: when a request carries **none** of those headers the credential digest is a constant, and such requests can coalesce with one another. That is deliberate. A caller presenting no credential has no per-caller billing identity to leak — every such request is answered under whatever single credential the deployment supplies, or is rejected alike. The isolation this buys is between callers the **upstream** can tell apart. + ## Session backup (proxy mode, opt-in) A belt-and-suspenders backup against CC's transcript regressions per [anthropics/claude-code#66734](https://github.com/anthropics/claude-code/issues/66734) (in-place transcript rewrite to a metadata-only stub) and [anthropics/claude-code#66486](https://github.com/anthropics/claude-code/issues/66486) (missing transcript on interactive sessions). When the proxy is in the path, every assistant message + observed tool result / user input is mirrored into a per-session JSONL file under user control, independent of CC's own transcript writer. CC's transcript remains canonical when it survives; the mirror is the recovery path when it doesn't. diff --git a/proxy/server.mjs b/proxy/server.mjs index a9caa4b8..1b9d3ecb 100644 --- a/proxy/server.mjs +++ b/proxy/server.mjs @@ -64,6 +64,201 @@ function debugLog(...args) { try { appendFileSync(path, line); } catch {} } +// ── Duplicate sidecar coalescing — threat-matrix row 31 ────────────────── +// +// CC issues one sidecar request TWICE, 6-25 ms apart on 47/47 measured +// pairs, with distinct upstream request-ids and two completed usage-log +// records: both sends are answered and both are charged (48,203 input-side +// tokens corpus-wide). Dropping the second is unavailable — two client +// requests are in flight and each is owed a response — so the only safe +// shape is ONE upstream call serving both callers. +// +// Four conditions, all required. The mid-session duplicate class, where a +// second send is a legitimate retry and suppressing it would leave a real +// request unanswered, fails on `nMsg` alone — that is the discriminator +// row 31 asked for: +// +// 1. exactly one message 3. byte-identical FORWARDED bodies, +// 2. no tools[] from the SAME caller +// 4. < 50 ms, first still in flight +// +// Why substituting one answer for the other is fidelity-safe, which is the +// objection that kept this parked: a request carrying no conversation +// history and no tools produces output that can enter no cached prefix, and +// CC issued the second send before it could have observed the first, so it +// already treats the two as interchangeable. +// +// Condition 3 is checked on the bytes we ACTUALLY send, after every +// extension has run — identical forwarded bodies is what makes the two +// upstream calls the same call — and it IS the map key: a full-length +// sha256 over the caller's credential digest and those bytes, so a hit +// already means byte-identical AND same-caller. Why the caller half is +// there: `coalesceIdentity` below. +const COALESCE_WINDOW_MS = 50; + +/** key (sha256 of credential digest + forwarded bytes) -> in-flight leader. */ +const inFlightSidecars = new Map(); + +// Entries normally leave on their leader's `close`. That is not guaranteed: +// a client that hangs without closing its socket, against an upstream that +// never answers, leaves one behind for as long as both hold. So the map is +// bounded by SWEEPING rather than trusting the event. +// +// The sweep needs no timer and no LRU because the window already bounds an +// entry's usefulness: an entry is only ever consulted inside COALESCE_WINDOW_MS, +// so one twice that old can never be hit again — the window check below would +// reject it. Deleting it is therefore invisible to the mechanism, and running +// the sweep at INSERT is what makes the map's size proportional to the +// coalesce-candidate requests of the last 100 ms rather than to the process's +// lifetime. +// +// Deleting only, never settling: `entry.done` is what an attached follower +// awaits, and resolving it here would answer for a leader still streaming. +// The leader's own `close` handler still settles if it ever fires. +export const COALESCE_SWEEP_AFTER_MS = COALESCE_WINDOW_MS * 2; + +function sweepInFlightSidecars(now = Date.now()) { + for (const [key, entry] of inFlightSidecars) { + if (now - entry.at >= COALESCE_SWEEP_AFTER_MS) inFlightSidecars.delete(key); + } +} + +// Exported for the bite: the map is module-private, and a bound that cannot +// be observed is a bound nothing can show red. +export function inFlightSidecarCount() { + return inFlightSidecars.size; +} + +// The key separates CALLERS as well as bytes, which is condition 3's other +// half. The session-start sidecar has a FIXED shape — same model, same +// max_tokens, one no-tools message — so on a shared proxy two different +// users' sends are byte-identical. Under a body-only key one user's +// in-flight call would answer the other's request: the leader's account is +// billed for both, a leader's 401 propagates to a follower holding valid +// credentials, and the leader's plan-tier / org context ships in bytes the +// follower reads. +// +// The credential set is DERIVED from SENSITIVE_HEADERS rather than restated +// beside it. A hand-copied list stays green the day a new credential header +// is added to that one, and the failure direction of deriving is fewer +// coalesces, never a shared one. `set-cookie` is a response header and is +// dropped. +// +// When a request carries none of them the digest is a constant, and that is +// deliberate rather than a hole: a caller presenting no credential has no +// per-caller billing identity to leak — every such request is answered under +// whatever single credential the deployment supplies, or 401s alike. The +// leak this closes is between callers the UPSTREAM can tell apart. +// +// Header names arrive lowercased from Node, but `reqCtx.headers` is handed +// to extensions to mutate, so the scan keys on the lowercased name — the +// invariant every spelling carries — not on the casing in hand. Entries are +// sorted so two identical header sets cannot digest differently on +// insertion order alone. +const COALESCE_IDENTITY_HEADERS = new Set( + [...SENSITIVE_HEADERS].filter((h) => h !== "set-cookie"), +); + +export function coalesceIdentity(headers) { + const material = []; + for (const [name, value] of Object.entries(headers || {})) { + const lower = name.toLowerCase(); + if (COALESCE_IDENTITY_HEADERS.has(lower)) material.push([lower, String(value)]); + } + material.sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)); + // A DIGEST, never the credential: the key's first 16 chars reach the debug + // log, and SENSITIVE_HEADERS exists so that file never holds one. + const h = createHash("sha256"); + for (const [name, value] of material) h.update(name).update("\0").update(value).update("\0"); + return h.digest("hex"); +} + +// Conditions 1 and 2 — the half that is a property of ONE request. 3 and 4 +// belong to a PAIR and are checked at the map hit. Exported for the bites: +// a predicate whose arms are only reachable through a live socket cannot be +// shown red on the case it was built for. +export function coalesceCandidate(parsed) { + if (!parsed || typeof parsed !== "object") return false; + if (!Array.isArray(parsed.messages) || parsed.messages.length !== 1) return false; + if (Array.isArray(parsed.tools) && parsed.tools.length > 0) return false; + return true; +} + +// One writable face over N client responses. `streamResponse` touches only +// write / once("drain") / end, and the non-streaming branch only +// writeHead / end, so this is the whole surface either needs. +// +// A follower may attach mid-stream, so the leader keeps every chunk it has +// written and replays it on attach — the alternative (buffering the whole +// response before writing any of it) would convert the leader's stream into +// a single delivery, which is a behaviour change to the streaming path for +// every coalesced request. +export function createFanOut(leaderRes) { + const sinks = [leaderRes]; + const replay = []; + let head = null; + let ended = false; + + const live = () => sinks.filter((r) => !r.writableEnded && !r.destroyed); + + return { + get sinkCount() { return live().length; }, + get writableEnded() { return live().length === 0; }, + writeHead(status, headers) { + head = { status, headers }; + for (const r of live()) r.writeHead(status, headers); + }, + // Returns false when the follower arrived too late to join — the caller + // has already been served in full and owes it nothing further. + attach(res) { + if (head) { + res.writeHead(head.status, head.headers); + for (const chunk of replay) res.write(chunk); + } + if (ended) { + res.end(); + return false; + } + sinks.push(res); + return true; + }, + write(chunk) { + replay.push(chunk); + let ok = true; + for (const r of live()) { + if (!r.write(chunk)) ok = false; + } + return ok; + }, + once(event, cb) { + if (event !== "drain") return; + const pending = live().filter((r) => r.writableNeedDrain); + if (pending.length === 0) { + setImmediate(cb); + return; + } + // The slowest attached client governs, and a client that goes away + // mid-drain must not hold the others: `close` counts as drained, or a + // follower hanging up would stall the leader's stream forever. + let left = pending.length; + const fire = () => { if (--left === 0) cb(); }; + for (const r of pending) { + r.once("drain", fire); + r.once("close", fire); + } + }, + end(data) { + if (data !== undefined) replay.push(data); + ended = true; + for (const r of live()) r.end(data); + }, + destroy(err) { + ended = true; + for (const r of live()) r.destroy(err); + }, + }; +} + function collectBody(req) { return new Promise((resolve, reject) => { const chunks = []; @@ -173,8 +368,18 @@ async function handleMessages(clientReq, clientRes) { // clientRes's close fires when the response is finished OR the connection is // destroyed, so pairing it with writableEnded separates the two: ended means // we answered, not-ended means the client hung up and the upstream should go. + // + // ONE MORE CALLER CAN BE WAITING. Under coalescing (gated off by default) + // `sink` becomes the leader's fan-out and the upstream call is serving + // followers as well, so "this client hung up" stops meaning "nobody needs + // the response". The fan-out's live-sink count is what answers that, and it + // is asked only when a fan-out is in play — on the plain path this guard is + // byte-for-byte the one above it. + let sink = clientRes; clientRes.on("close", () => { - if (!clientRes.writableEnded) abortController.abort(); + if (clientRes.writableEnded) return; // answered; nothing to free + if (sink !== clientRes && sink.sinkCount > 0) return; // followers still reading + abortController.abort(); }); const pre = await preForward(clientReq, clientRes, abortController, extSnapshot, "messages"); @@ -187,6 +392,48 @@ async function handleMessages(clientReq, clientRes) { } const { parsed, forwardBody, headers, meta } = pre; + // Row 31. Gated OFF by default: the mechanism ships with its bites, and + // enabling it is a separate, declared act (ship-proxy-change step 4b). + const coalesceKey = process.env.CACHE_FIX_COALESCE_SIDECAR === "1" && coalesceCandidate(parsed) + ? createHash("sha256") + .update(coalesceIdentity(headers)).update("\0") + .update(forwardBody) + .digest("hex") + : null; + + if (coalesceKey) { + const leader = inFlightSidecars.get(coalesceKey); + // Condition 3 IS the key: a full-length sha256 of the forwarded bytes, + // so a map hit already means byte-identical. A second `Buffer.equals` + // beside it was written here first and removed after the mutation proof + // — disabling it left every arm green, because differing bodies produce + // a different key and never reach the compare. Its only falsifying input + // is a sha256 collision, which makes it an unprovable predicate wearing + // a check's clothes. Condition 4, the window, is what remains here. + if (leader && Date.now() - leader.at < COALESCE_WINDOW_MS) { + debugLog("[PROXY] coalescing duplicate sidecar into in-flight request", + "key:", coalesceKey.slice(0, 16), "ageMs:", Date.now() - leader.at); + if (leader.fanOut.attach(clientRes)) await leader.done; + return; + } + sweepInFlightSidecars(); + let settle; + const entry = { + at: Date.now(), + fanOut: createFanOut(clientRes), + done: new Promise((r) => { settle = r; }), + }; + entry.settle = settle; + inFlightSidecars.set(coalesceKey, entry); + sink = entry.fanOut; + // The map entry outlives neither the request nor an early throw: every + // exit below runs through this. + clientRes.on("close", () => { + if (inFlightSidecars.get(coalesceKey) === entry) inFlightSidecars.delete(coalesceKey); + entry.settle(); + }); + } + const requestedModel = parsed?.model || null; let upstreamRes, responseHeaders, statusCode, upstreamConnectionId; @@ -204,8 +451,8 @@ async function handleMessages(clientReq, clientRes) { } catch (err) { debugLog("[PROXY] forwardRequest error:", err.message); if (abortController.signal.aborted) return; - clientRes.writeHead(502, { "content-type": "application/json" }); - clientRes.end(JSON.stringify({ error: "upstream_error", message: err.message })); + sink.writeHead(502, { "content-type": "application/json" }); + sink.end(JSON.stringify({ error: "upstream_error", message: err.message })); return; } @@ -241,35 +488,41 @@ async function handleMessages(clientReq, clientRes) { if (responseBody) { const resCtx = { status: statusCode, headers: responseHeaders, body: responseBody, meta }; await runOnResponse(resCtx, extSnapshot); - clientRes.writeHead(statusCode, resCtx.headers); - clientRes.end(JSON.stringify(resCtx.body)); + sink.writeHead(statusCode, resCtx.headers); + sink.end(JSON.stringify(resCtx.body)); } else { - clientRes.writeHead(statusCode, responseHeaders); - clientRes.end(rawResponse); + sink.writeHead(statusCode, responseHeaders); + sink.end(rawResponse); } } else { - clientRes.writeHead(statusCode, responseHeaders); - clientRes.end(rawResponse); + sink.writeHead(statusCode, responseHeaders); + sink.end(rawResponse); } return; } - clientRes.writeHead(statusCode, responseHeaders); + sink.writeHead(statusCode, responseHeaders); const telemetry = createTelemetryRecord(); telemetry.requestedModel = requestedModel; upstreamRes.on("error", (err) => { - if (!clientRes.writableEnded) { - clientRes.destroy(err); + if (!sink.writableEnded) { + sink.destroy(err); } }); try { - await streamResponse(upstreamRes, clientRes, telemetry, extSnapshot, meta, responseHeaders); + // Fan-out sits at the RESPONSE WRITER, never at the upstream reader: the + // extension pass and the telemetry record run exactly once, so both + // callers receive byte-identical post-pipeline output. Tee-ing the raw + // upstream instead would hand the follower unmutated bytes while the + // leader got the pipeline's — a fidelity split, and fidelity outranks + // cache here. + await streamResponse(upstreamRes, sink, telemetry, extSnapshot, meta, responseHeaders); } catch (err) { - if (!clientRes.writableEnded) { - clientRes.destroy(err); + if (!sink.writableEnded) { + sink.destroy(err); } } } diff --git a/test/duplicate-coalesce.test.mjs b/test/duplicate-coalesce.test.mjs new file mode 100644 index 00000000..fdd552fe --- /dev/null +++ b/test/duplicate-coalesce.test.mjs @@ -0,0 +1,520 @@ +// Threat-matrix row 31: CC issues one sidecar request TWICE, 6-25 ms apart, +// with distinct upstream request-ids and two completed usage-log records — +// both answered, both charged. The mitigation coalesces the pair into ONE +// upstream call serving both callers. +// +// These bites exercise the predicate AT THE WIRE, through a real proxy +// instance against a real (local) upstream that counts what it received, +// because the defect is a count of upstream calls and nothing below that +// altitude can observe it. The arms MUST DIFFER: an assertion that only +// showed "one call" for the coalescing case would pass equally against a +// build that coalesced everything, which is the over-reach this predicate +// exists to prevent — so the mid-session arm asserting TWO calls is the +// discriminating half, not decoration. + +import { tmpDir } from "../tools/tmpdir.mjs"; +import { describe, it, before, after } from "node:test"; +import assert from "node:assert/strict"; +import http from "node:http"; +import { rm } from "node:fs/promises"; +import { + startProxy, coalesceCandidate, coalesceIdentity, createFanOut, + inFlightSidecarCount, COALESCE_SWEEP_AFTER_MS, +} from "../proxy/server.mjs"; + +// Two callers the UPSTREAM can tell apart. Synthetic by construction — the +// point is only that the two strings differ, so nothing here needs to look +// like a real credential. +const TENANT_A = { "x-api-key": "tenant-a-credential" }; +const TENANT_B = { "x-api-key": "tenant-b-credential" }; + +function clientRequest(port, body, extraHeaders = {}) { + return new Promise((resolve, reject) => { + const data = JSON.stringify(body); + const req = http.request( + { + hostname: "127.0.0.1", + port, + path: "/v1/messages", + method: "POST", + headers: { "content-type": "application/json", ...extraHeaders }, + }, + (res) => { + const chunks = []; + res.on("data", (c) => chunks.push(c)); + res.on("end", () => resolve({ status: res.statusCode, body: Buffer.concat(chunks).toString() })); + }, + ); + req.on("error", reject); + req.end(data); + }); +} + +// A caller whose hang-up we choose the moment of. `clientRequest` above reads +// to the end, which never reaches the abort guard — the guard only fires on a +// response that closes UNANSWERED, so producing that state deliberately is the +// only way to exercise it. +function openAndHangUpOnCommand(port, body) { + let hangUp; + const done = new Promise((resolve) => { + const req = http.request( + { + hostname: "127.0.0.1", + port, + path: "/v1/messages", + method: "POST", + headers: { "content-type": "application/json" }, + }, + (res) => { + res.on("data", () => {}); + res.on("end", () => resolve("ended")); + }, + ); + // BOTH, and "close" is the one that fires: `req.destroy()` with no error + // argument emits no "error" at all, so an error-only settle leaves this + // promise pending forever and the case times out instead of asserting. + req.on("error", () => resolve("hung-up")); + req.on("close", () => resolve("hung-up")); + hangUp = () => req.destroy(); + req.end(JSON.stringify(body)); + }); + return { done, hangUp: () => hangUp() }; +} + +// Holds the response open long enough that a duplicate arriving inside the +// 50 ms window finds the first still IN FLIGHT — condition 4. Without the +// hold the first call would complete before the second arrived and the +// coalescing arm would pass for the wrong reason. +function slowSseUpstream(counter, holdMs = 120) { + return http.createServer((req, res) => { + const chunks = []; + req.on("data", (c) => chunks.push(c)); + req.on("end", () => { + counter.calls += 1; + counter.bodies.push(Buffer.concat(chunks).toString()); + res.writeHead(200, { "content-type": "text/event-stream" }); + res.write('data: {"type":"message_start","message":{"model":"claude-haiku-4-5","usage":{}}}\n\n'); + setTimeout(() => { + res.write('data: {"type":"message_stop"}\n\n'); + res.write("data: [DONE]\n\n"); + res.end(); + }, holdMs); + }); + }); +} + +const SIDECAR = { + model: "claude-haiku-4-5", + max_tokens: 32000, + stream: true, + messages: [{ role: "user", content: [{ type: "text", text: "x".repeat(337) }] }], +}; + +const MID_SESSION = { + ...SIDECAR, + messages: [ + { role: "user", content: [{ type: "text", text: "first" }] }, + { role: "assistant", content: [{ type: "text", text: "reply" }] }, + { role: "user", content: [{ type: "text", text: "second" }] }, + ], +}; + +describe("row 31 — the structural half of the predicate (conditions 1 and 2)", () => { + it("accepts a single-message request carrying no tools", () => { + assert.equal(coalesceCandidate(SIDECAR), true); + }); + + it("REJECTS a mid-session request — nMsg alone is the discriminator the row asked for", () => { + assert.equal(coalesceCandidate(MID_SESSION), false); + }); + + it("REJECTS a single-message request that carries tools", () => { + assert.equal(coalesceCandidate({ ...SIDECAR, tools: [{ name: "Bash" }] }), false); + }); + + it("treats an EMPTY tools array as no tools — the measured request carried 0", () => { + assert.equal(coalesceCandidate({ ...SIDECAR, tools: [] }), true); + }); + + it("rejects a body with no messages array at all", () => { + assert.equal(coalesceCandidate({ model: "x" }), false); + assert.equal(coalesceCandidate(null), false); + }); +}); + +describe("row 31 — the key separates CALLERS, not only bytes", () => { + // The shared-proxy leak the coalescing key has to close: the session-start + // sidecar has a FIXED shape, so two different users' sends are byte- + // identical. A body-only key lets one user's in-flight call answer the + // other's request — leader's account billed for both, leader's 401 + // propagated to a follower holding valid credentials, leader's plan-tier + // context in bytes the follower reads. + + it("different credentials produce different identities", () => { + assert.notEqual(coalesceIdentity(TENANT_A), coalesceIdentity(TENANT_B)); + }); + + it("the same credential produces the same identity — coalescing must still happen", () => { + assert.equal(coalesceIdentity({ ...TENANT_A }), coalesceIdentity({ ...TENANT_A })); + }); + + it("separates on `authorization` too, not just `x-api-key`", () => { + assert.notEqual( + coalesceIdentity({ authorization: "Bearer aaa" }), + coalesceIdentity({ authorization: "Bearer bbb" }), + ); + }); + + it("keys on the LOWERCASED header name — an extension may have added its own casing", () => { + // Node lowercases what arrives on the wire, but `reqCtx.headers` is + // handed to extensions to mutate, and an extension setting `X-Api-Key` + // must not read as a different caller from one setting `x-api-key`. + assert.equal(coalesceIdentity({ "X-Api-Key": "same" }), coalesceIdentity({ "x-api-key": "same" })); + }); + + it("ignores headers that carry no caller identity", () => { + assert.equal( + coalesceIdentity({ ...TENANT_A, "content-type": "application/json" }), + coalesceIdentity({ ...TENANT_A, "content-type": "text/plain", "x-request-id": "abc" }), + ); + }); + + it("is a CONSTANT when the request carries no credential at all", () => { + // Deliberate, not a hole. A caller presenting no credential has no + // per-caller billing identity to leak: every such request is answered + // under whatever single credential the deployment supplies, or 401s + // alike. The leak the key closes is between callers the upstream can + // tell apart. + assert.equal(coalesceIdentity({}), coalesceIdentity({ "content-type": "application/json" })); + }); + + it("never carries the credential itself — the identity is a digest", () => { + // The key's first 16 chars are written to the debug log. A key built by + // concatenating the raw credential would put it on disk, which is what + // SENSITIVE_HEADERS exists upstream of this to prevent. + assert.doesNotMatch(coalesceIdentity(TENANT_A), /tenant-a-credential/); + assert.match(coalesceIdentity(TENANT_A), /^[0-9a-f]{64}$/); + }); +}); + +describe("row 31 — the fan-out writable serves every attached caller", () => { + function fakeRes() { + return { + writableEnded: false, destroyed: false, writableNeedDrain: false, + head: null, chunks: [], ended: false, + writeHead(status, headers) { this.head = { status, headers }; }, + write(c) { this.chunks.push(String(c)); return true; }, + end(c) { if (c !== undefined) this.chunks.push(String(c)); this.ended = true; this.writableEnded = true; }, + destroy() { this.destroyed = true; }, + once() {}, + }; + } + + it("a follower attaching MID-STREAM receives the chunks already written", () => { + const leader = fakeRes(); + const fan = createFanOut(leader); + fan.writeHead(200, { "content-type": "text/event-stream" }); + fan.write("data: one\n\n"); + + const follower = fakeRes(); + assert.equal(fan.attach(follower), true); + + fan.write("data: two\n\n"); + fan.end(); + + assert.deepEqual(leader.chunks, ["data: one\n\n", "data: two\n\n"]); + assert.deepEqual(follower.chunks, ["data: one\n\n", "data: two\n\n"], + "the follower must receive the whole response, not only what came after it attached"); + assert.deepEqual(follower.head, { status: 200, headers: { "content-type": "text/event-stream" } }); + assert.equal(follower.ended, true); + }); + + it("a follower attaching AFTER the response ended is served in full and reports not-joined", () => { + const leader = fakeRes(); + const fan = createFanOut(leader); + fan.writeHead(200, {}); + fan.write("data: one\n\n"); + fan.end(); + + const late = fakeRes(); + assert.equal(fan.attach(late), false, "a late follower must not be added to the live set"); + assert.deepEqual(late.chunks, ["data: one\n\n"]); + assert.equal(late.ended, true, "it is still owed a complete response"); + }); + + it("a leader whose own client hung up keeps writing to its followers", () => { + const leader = fakeRes(); + const fan = createFanOut(leader); + fan.writeHead(200, {}); + const follower = fakeRes(); + fan.attach(follower); + + leader.destroyed = true; // the leader's client goes away mid-stream + fan.write("data: after\n\n"); + + assert.equal(fan.writableEnded, false, "someone is still listening"); + assert.deepEqual(follower.chunks, ["data: after\n\n"]); + assert.deepEqual(leader.chunks, [], "nothing is written to a dead socket"); + }); +}); + +describe("row 31 at the wire — the upstream call COUNT is the defect", () => { + let handle, upstream, counter, extDir; + + before(async () => { + extDir = await tmpDir("coalesce-ext-"); + counter = { calls: 0, bodies: [] }; + upstream = slowSseUpstream(counter); + await new Promise((r) => upstream.listen(0, "127.0.0.1", r)); + process.env.CACHE_FIX_PROXY_UPSTREAM = `http://127.0.0.1:${upstream.address().port}`; + process.env.CACHE_FIX_COALESCE_SIDECAR = "1"; + handle = await startProxy({ port: 0, watch: false, extensionsDir: extDir }); + }); + + after(async () => { + await handle.close(); + await new Promise((r) => upstream.close(r)); + delete process.env.CACHE_FIX_PROXY_UPSTREAM; + delete process.env.CACHE_FIX_COALESCE_SIDECAR; + await rm(extDir, { recursive: true, force: true }); + }); + + it("all four conditions: ONE upstream call, BOTH callers answered", async () => { + counter.calls = 0; + const a = clientRequest(handle.port, SIDECAR); + await new Promise((r) => setTimeout(r, 15)); // inside the 50 ms window + const b = clientRequest(handle.port, SIDECAR); + const [ra, rb] = await Promise.all([a, b]); + + assert.equal(counter.calls, 1, "the duplicate must not reach upstream"); + assert.equal(ra.status, 200); + assert.equal(rb.status, 200); + assert.equal(ra.body, rb.body, "both callers receive byte-identical output"); + assert.ok(ra.body.includes("message_stop"), "and it is the COMPLETE response, not a truncated replay"); + }); + + it("the leader's client hangs up mid-stream: the FOLLOWER is still served in full", async () => { + // THE ABORT GUARD'S POLARITY, and nothing else in this file reaches it. + // On the plain path "this response closed without being answered" means + // nobody is left to serve, so freeing the upstream call is right. Under + // coalescing it stops meaning that: the leader's fan-out is carrying a + // follower, and aborting cuts the call THAT caller is waiting on. + // + // Order matters here and the first cut had it wrong: hanging up before + // the follower arrives removes the leader from the map, the pair never + // coalesces, and the arm passes for the wrong reason. The premise + // assertion below is what pins that. + // + // Measured both ways before this landed. Against the naive guard + // (`if (!sink.writableEnded) abortController.abort()`) this case goes red + // and the other fifteen stay green; against the guard as written all + // sixteen pass. The red is a TIMEOUT rather than a truncated body, which + // is the more useful fact: aborting the upstream call leaves the fan-out + // with nothing to end, so the follower is not cut short — it hangs until + // its own timeout, holding a live session open on a request that will + // never answer. + counter.calls = 0; + const leader = openAndHangUpOnCommand(handle.port, SIDECAR); + await new Promise((r) => setTimeout(r, 15)); // inside the 50 ms window + const follower = clientRequest(handle.port, SIDECAR); + await new Promise((r) => setTimeout(r, 20)); // follower attached; upstream still holding + leader.hangUp(); + await leader.done; + const rb = await follower; + + assert.equal(counter.calls, 1, + "premise: the pair did not coalesce, so there was no follower on the leader's call and this arm tests nothing"); + assert.equal(rb.status, 200); + assert.ok(rb.body.includes("message_stop"), + `the follower was cut off when the LEADER's client left: ${JSON.stringify(rb.body)}`); + }); + + it("TWO TENANTS, byte-identical sidecars inside the window: TWO upstream calls", async () => { + // The shared-proxy arm. Every one of the four conditions holds and the + // pair must STILL not coalesce, because the callers are different. + counter.calls = 0; + const a = clientRequest(handle.port, SIDECAR, TENANT_A); + await new Promise((r) => setTimeout(r, 15)); // inside the 50 ms window + const b = clientRequest(handle.port, SIDECAR, TENANT_B); + await Promise.all([a, b]); + + assert.equal(counter.calls, 2, + "one tenant's in-flight call must never answer another tenant's request"); + }); + + it("ONE tenant's duplicate still coalesces: ONE upstream call", async () => { + // The other half of the pair, and it is not decoration: the arm above + // passes equally against a build that stopped coalescing altogether, or + // one whose key picked up something per-request. This is what pins that + // the identity is per-CALLER and not per-REQUEST. + counter.calls = 0; + const a = clientRequest(handle.port, SIDECAR, TENANT_A); + await new Promise((r) => setTimeout(r, 15)); + const b = clientRequest(handle.port, SIDECAR, TENANT_A); + const [ra, rb] = await Promise.all([a, b]); + + assert.equal(counter.calls, 1, "the same caller's duplicate is exactly what row 31 coalesces"); + assert.equal(ra.body, rb.body); + }); + + it("mid-session pair (nMsg > 1): TWO upstream calls, unchanged", async () => { + counter.calls = 0; + const a = clientRequest(handle.port, MID_SESSION); + await new Promise((r) => setTimeout(r, 15)); + const b = clientRequest(handle.port, MID_SESSION); + await Promise.all([a, b]); + + assert.equal(counter.calls, 2, + "a mid-session duplicate is a legitimate retry — suppressing it would leave a real request unanswered"); + }); + + it("three of four conditions (tools present): TWO upstream calls", async () => { + counter.calls = 0; + const withTools = { ...SIDECAR, tools: [{ name: "Bash", input_schema: {} }] }; + const a = clientRequest(handle.port, withTools); + await new Promise((r) => setTimeout(r, 15)); + const b = clientRequest(handle.port, withTools); + await Promise.all([a, b]); + + assert.equal(counter.calls, 2, "failing any one condition must not coalesce"); + }); + + it("three of four conditions (still in flight, but PAST the 50 ms window): TWO upstream calls", async () => { + // The second send must arrive while the first is STILL IN FLIGHT (the + // upstream holds 120 ms) but outside the window, or this arm proves + // nothing about condition 4. The first version awaited both requests + // sequentially, so the leader had already left the map and the window + // check was never reached — disabling the window left it green, which + // is how the gap was found. + counter.calls = 0; + const a = clientRequest(handle.port, SIDECAR); + await new Promise((r) => setTimeout(r, 80)); + const b = clientRequest(handle.port, SIDECAR); + await Promise.all([a, b]); + + assert.equal(counter.calls, 2, "past the window the pair is not a duplicate send"); + }); + + it("a sequential repeat (first already completed) is not coalesced", async () => { + counter.calls = 0; + await clientRequest(handle.port, SIDECAR); + await clientRequest(handle.port, SIDECAR); + + assert.equal(counter.calls, 2, "the leader must not outlive its own request"); + }); + + it("differing bodies inside the window: TWO upstream calls", async () => { + counter.calls = 0; + const a = clientRequest(handle.port, SIDECAR); + await new Promise((r) => setTimeout(r, 15)); + const b = clientRequest(handle.port, { ...SIDECAR, max_tokens: 16000 }); + await Promise.all([a, b]); + + // What this establishes, stated precisely because the first version of + // this arm claimed more: differing forwarded bytes produce a different + // KEY, so the pair never meets. It does not exercise a separate + // byte-compare, and the mutation proof is what showed that — disabling + // one left this arm green. + assert.equal(counter.calls, 2, "differing forwarded bytes never share a coalescing key"); + }); +}); + +describe("row 31 — the in-flight map is BOUNDED, not trusting `close`", () => { + // Entries normally leave on their leader's `close`. A client that hangs + // without closing its socket, against an upstream that never answers, + // leaves one behind for as long as both hold — so the bound cannot rest on + // that event. This arm builds exactly that state: an upstream that accepts + // the request and never responds, and a client that never hangs up. + const HANG = "HANG-FOREVER"; + let handle, upstream, held, extDir; + + before(async () => { + extDir = await tmpDir("coalesce-bound-ext-"); + held = []; + upstream = http.createServer((req, res) => { + const chunks = []; + req.on("data", (c) => chunks.push(c)); + req.on("end", () => { + if (Buffer.concat(chunks).toString().includes(HANG)) { held.push(res); return; } + res.writeHead(200, { "content-type": "text/event-stream" }); + res.write('data: {"type":"message_stop"}\n\n'); + res.write("data: [DONE]\n\n"); + res.end(); + }); + }); + await new Promise((r) => upstream.listen(0, "127.0.0.1", r)); + process.env.CACHE_FIX_PROXY_UPSTREAM = `http://127.0.0.1:${upstream.address().port}`; + process.env.CACHE_FIX_COALESCE_SIDECAR = "1"; + handle = await startProxy({ port: 0, watch: false, extensionsDir: extDir }); + }); + + after(async () => { + for (const r of held) { try { r.destroy(); } catch {} } + await handle.close(); + await new Promise((r) => upstream.close(r)); + delete process.env.CACHE_FIX_PROXY_UPSTREAM; + delete process.env.CACHE_FIX_COALESCE_SIDECAR; + await rm(extDir, { recursive: true, force: true }); + }); + + it("a stale leader whose client never closes is swept by the next insert", async () => { + const stuckBody = { ...SIDECAR, messages: [{ role: "user", content: [{ type: "text", text: HANG }] }] }; + const stuck = http.request( + { hostname: "127.0.0.1", port: handle.port, path: "/v1/messages", method: "POST", + headers: { "content-type": "application/json" } }, + (res) => res.on("data", () => {}), + ); + stuck.on("error", () => {}); + stuck.end(JSON.stringify(stuckBody)); + + await new Promise((r) => setTimeout(r, 40)); + assert.equal(inFlightSidecarCount(), 1, + "premise: the stuck leader never entered the map, so this arm tests nothing"); + + await new Promise((r) => setTimeout(r, COALESCE_SWEEP_AFTER_MS + 20)); + assert.equal(inFlightSidecarCount(), 1, + "premise: nothing sweeps on a timer — the entry survives until an insert, which is what the next line exercises"); + + // A DIFFERENT body, so it takes its own key and does not coalesce with + // the stuck one. Its insert is what runs the sweep. + await clientRequest(handle.port, { ...SIDECAR, max_tokens: 16000 }); + await new Promise((r) => setTimeout(r, 20)); // let the second leader's own close land + + assert.equal(inFlightSidecarCount(), 0, + "the stale leader outlived the window it could be hit in — the map grows with process lifetime"); + + stuck.destroy(); + }); +}); + +describe("row 31 — the gate is OFF by default", () => { + let handle, upstream, counter, extDir; + + before(async () => { + extDir = await tmpDir("coalesce-off-ext-"); + counter = { calls: 0, bodies: [] }; + upstream = slowSseUpstream(counter); + await new Promise((r) => upstream.listen(0, "127.0.0.1", r)); + process.env.CACHE_FIX_PROXY_UPSTREAM = `http://127.0.0.1:${upstream.address().port}`; + delete process.env.CACHE_FIX_COALESCE_SIDECAR; + handle = await startProxy({ port: 0, watch: false, extensionsDir: extDir }); + }); + + after(async () => { + await handle.close(); + await new Promise((r) => upstream.close(r)); + delete process.env.CACHE_FIX_PROXY_UPSTREAM; + await rm(extDir, { recursive: true, force: true }); + }); + + it("without the gate the duplicate still reaches upstream — the pre-fix behaviour, pinned", async () => { + counter.calls = 0; + const a = clientRequest(handle.port, SIDECAR); + await new Promise((r) => setTimeout(r, 15)); + const b = clientRequest(handle.port, SIDECAR); + await Promise.all([a, b]); + + assert.equal(counter.calls, 2, + "this is the RED baseline: the same input under the shipped-but-disabled build double-bills"); + }); +}); diff --git a/tools/tmpdir.mjs b/tools/tmpdir.mjs new file mode 100644 index 00000000..12f55032 --- /dev/null +++ b/tools/tmpdir.mjs @@ -0,0 +1,201 @@ +// The one place this repo creates temporary directories. +// +// DEFINITION (what "correct" means here, so the checks below are not written +// from the implementation): a process that uses this helper leaves NOTHING +// under the OS temp root once it has exited. Not "usually", and not "when the +// happy path is taken" — the guarantee holds for a run that throws, a run that +// calls process.exit(), and a run that is terminated by SIGINT/SIGTERM/SIGHUP. +// The cases it cannot hold for are the ones that run no code at all: SIGKILL, +// and a V8 heap-limit failure, which calls abort() (SIGABRT, exit 134) without +// running a single exit handler. That residue is what gate-live's leftover +// signal exists to report, and the second case is not hypothetical — it is +// exactly what the `tmpLeftovers` guard was reporting for two days while this +// header named only SIGKILL and sent every reader looking for a kill that never +// happened (measured 2026-08-14: two aborted children per full suite run, +// manufactured by test/gate-live-rowpins.test.mjs's deliberate 8 MB-cap crash; +// that file now gives them a private TMPDIR, so the residue lands inside the +// suite's own scratch). A leftover root is therefore evidence that some child +// DIED HARD — which is a finding about that child, not about this module. +// +// WHY THIS EXISTS. Measured 2026-08-08: /tmp here is a 31 GB tmpfs and it +// reached 100% with 31,108 top-level directories — 7,024 `fixture-verd*`, +// ~8,000 `bt-*`, plus `census-*`, `harvest-*`, `verdict-*`, `ledger-*`, +// `mitigation-output-*` and more. Every one was an `mkdtemp` whose creator +// never removed it. The ENOSPC then broke unrelated tooling machine-wide while +// the test suite stayed GREEN — the silent-failure class — and it produced five +// consecutive runs of ONE commit returning 0, 3, 95, 525 and 528 failures, +// which read as a broken build and was first misdiagnosed as concurrency +// (docs/dev-loop.md, "A failure count that swings by hundreds"). +// +// WHY A PER-RUN PARENT rather than per-call cleanup. Per-call cleanup is what +// the leaking sites already tried: several tools do `rm(scratch)` on the happy +// path and skip it on every throw, and `fixture-verdict-identity.mjs` had a +// `finally` that restored env vars and forgot the directory. One parent per +// process, removed once at exit, makes the guarantee independent of how many +// call sites there are and of which of them remembered — a call site can only +// opt IN to the leak now, by not using this module. `test/no-raw-mkdtemp.test.mjs` +// is the writer-side guard that keeps that from happening quietly. +// +// NOT A REAPER, deliberately. This module never deletes anything it did not +// create in this process. A stale directory from someone else's run may belong +// to a run that is still going (a long replay legitimately outlives an hour), +// and a helper that swept the temp root on startup would be a destructive +// sweep racing every concurrent lane on this machine. Reporting is gate-live's +// job (`staleRunRoots` below); deleting stays a human decision. +import { mkdtempSync, rmSync, readdirSync, statSync } from "node:fs"; +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +// Every directory this repo creates under the temp root is a child of a parent +// carrying this prefix, which is what makes the leftover scan below a closed +// question rather than a list of prefixes someone has to remember to extend. +// +// The name carries the creating PID (`cache-fix-run--XXXXXX`) so the scan +// can tell a run that DIED from one that is merely slow. Without it the age +// threshold alone would fire on gate-live's own long replay children — a check +// firing on legitimate work, which trains its reader to ignore red. +export const RUN_ROOT_PREFIX = "cache-fix-run-"; +const RUN_ROOT_RE = /^cache-fix-run-(\d+)-/; + +let runRoot = null; +let removed = false; + +// Cleanup must not throw: it runs from an `exit` handler, where the only thing +// a throw can accomplish is masking the real result the process was about to +// report. The third answer belongs to the caller, not to teardown. +function removeRunRoot() { + if (!runRoot || removed) return; + removed = true; + try { + rmSync(runRoot, { recursive: true, force: true }); + } catch { + // Nothing useful to do at exit time. + } +} + +function ensureRunRoot() { + if (runRoot) return runRoot; + runRoot = mkdtempSync(join(tmpdir(), `${RUN_ROOT_PREFIX}${process.pid}-`)); + removed = false; + + // `exit` covers the ordinary paths AND the crash paths: node runs exit + // handlers after an uncaught exception and after an unhandled rejection, so + // "the run threw" needs no separate arm. It must be synchronous — hence + // rmSync — because async work at exit time never runs. + process.on("exit", removeRunRoot); + + // Signals are the arm `exit` does NOT cover, and they are not hypothetical: + // gate-live and harvest run under systemd timers, and systemd stops a unit + // with SIGTERM. `once` removes the listener before invoking it, so the + // explicit exit below is the only termination path — we re-exit with the + // conventional 128+signal code rather than re-raising, because that is + // predictable regardless of what else has attached a listener. + const SIGNAL_EXIT = { SIGINT: 130, SIGTERM: 143, SIGHUP: 129 }; + for (const [sig, code] of Object.entries(SIGNAL_EXIT)) { + process.once(sig, () => { + removeRunRoot(); + process.exit(code); + }); + } + return runRoot; +} + +/** + * Async temp directory, inside this process's run root. + * Replaces `await mkdtemp(join(tmpdir(), prefix))`. + * @param {string} prefix e.g. "cache-fix-replay-" + * @returns {Promise} absolute path to a fresh directory + */ +export async function tmpDir(prefix) { + return mkdtemp(join(ensureRunRoot(), prefix)); +} + +/** + * Sync temp directory, inside this process's run root. + * Replaces `mkdtempSync(join(tmpdir(), prefix))`. + * @param {string} prefix e.g. "bt-" + * @returns {string} absolute path to a fresh directory + */ +export function tmpDirSync(prefix) { + return mkdtempSync(join(ensureRunRoot(), prefix)); +} + +/** + * This process's run root, or null if nothing has been created yet. Exported + * for the tests that assert containment; callers have no reason to want it. + */ +export function currentRunRoot() { + return runRoot; +} + +/** + * Remove this process's run root now, rather than at exit. For a long-lived + * process that wants its scratch back mid-run; idempotent, and the exit + * handler stays registered so a later tmpDir() call is still covered. + */ +export function cleanupRunRoot() { + removeRunRoot(); + runRoot = null; +} + +// `kill(pid, 0)` probes existence without signalling. EPERM means the process +// exists and belongs to someone else, which still counts as alive. +function defaultIsAlive(pid) { + try { + process.kill(pid, 0); + return true; + } catch (err) { + return err.code === "EPERM"; + } +} + +/** + * Run roots left behind by OTHER runs — the SIGKILL residue, and the tell that + * a call site has stopped using this module. Reads only; deletes nothing. + * + * @param {object} [opts] + * @param {number} [opts.olderThanMs] age threshold, default 1 hour. A run root + * younger than this may well belong to a run that is still going, so it is + * not evidence of anything. + * @param {string} [opts.root] temp root to scan, default os.tmpdir(). + * @param {number} [opts.now] clock injection point, so the test does not have + * to sleep an hour to exercise the threshold. + * @param {(pid: number) => boolean} [opts.isAlive] liveness probe, injectable + * for the same reason. + * @returns {{count: number, dirs: string[], scanned: boolean, reason: string|null}} + * `scanned: false` with a reason is the third answer — the temp root could + * not be read, which is neither clean nor dirty and must not be reported as + * a count of zero. + */ +export function staleRunRoots({ + olderThanMs = 60 * 60 * 1000, + root = tmpdir(), + now = Date.now(), + isAlive = defaultIsAlive, +} = {}) { + let entries; + try { + entries = readdirSync(root, { withFileTypes: true }); + } catch (err) { + return { count: 0, dirs: [], scanned: false, reason: `cannot read ${root}: ${err.message}` }; + } + const dirs = []; + for (const e of entries) { + if (!e.isDirectory() || !e.name.startsWith(RUN_ROOT_PREFIX)) continue; + const full = join(root, e.name); + if (full === runRoot) continue; // our own, still in use + // A run that is still going owns its directory however old it is: a sweep + // over a large corpus legitimately outlives the threshold. + const pid = RUN_ROOT_RE.exec(e.name)?.[1]; + if (pid && isAlive(Number(pid))) continue; + try { + if (now - statSync(full).mtimeMs >= olderThanMs) dirs.push(full); + } catch { + // Vanished between readdir and stat — a concurrent run cleaning up after + // itself, which is the healthy case and not a finding. + } + } + dirs.sort(); + return { count: dirs.length, dirs, scanned: true, reason: null }; +}