diff --git a/CHANGELOG.md b/CHANGELOG.md index 13200ce..98e1ac1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,17 @@ match exactly, or the workflow fails the publish. quarantine, and a target project's configured retries count each test's final attempt as its outcome. +- **The pipeline always terminates, fails locally, and shows its cost.** Every + external await is now time-boxed (model calls — with one backed-off retry on + a transient failure — shell commands, Playwright invocations, the rac graph + export, git); a runner exception counts as a failed fidelity attempt with + the reason on the verdict instead of aborting the gate; one capability's + failure in a scoped run no longer discards its siblings' results; + observations are capped per turn so the transcript can't grow without + bound. The bundled adapters surface provider token usage (shown in the QA + summary), and `--verbose` streams a per-turn audit trail (tool calls, + errors, model latency) to stderr. + ## 2026.07.1 — the "any model" release The release that makes **bring-your-own-model** mean *any* model — and proves Proofkeeper on itself. Everything since the first cut: diff --git a/lore-proofkeeper/designs/design-drive-resilience.md b/lore-proofkeeper/designs/design-drive-resilience.md new file mode 100644 index 0000000..c82e6cc --- /dev/null +++ b/lore-proofkeeper/designs/design-drive-resilience.md @@ -0,0 +1,98 @@ +--- +schema_version: 1 +id: PK-KWFWJNJ7NHN4 +type: design +--- +# Drive Resilience — Bounded Time, Isolated Failure, Visible Cost + +## Context + +A review found the pipeline had no wall-clock bounds anywhere (`grep` for +timeout/retry found only the extension loader), all-or-nothing failure in the +scoped pool and the fidelity gate, unbounded transcript growth, and zero cost +or audit visibility. This design adds the resilience layer without changing +any verdict semantics. + +## User Need + +An operator running Proofkeeper unattended — in CI or over a large scoped +change — needs the run to always terminate, to lose only the failing +capability when something breaks, and to be able to see afterwards what the +agent did and what it cost. + +## Design + +- **Time-box every external await.** The drive wraps `model.complete` in a + timeout (default 2 min, `modelTimeoutMs`) with one backed-off retry + (`modelRetryBackoffMs`, default 2 s); `runCommand` gains + `timeout`/`maxBuffer` (2 min / 16 MB), mirrored in the emitted spec's inline + helper so record and replay stay in agreement; the Playwright invocation + (10 min, `timeoutMs` option), the rac export (2 min), and the git diff + (1 min) get `execFile` timeouts. +- **Failure is local.** The fidelity gate wraps each attempt: a runner + exception is a failed attempt recorded on `FidelityVerdict.errors`, so "the + test failed" and "the run broke" are distinguishable and the gate always + completes. The scoped pool wraps the whole per-capability `runQa` in + try/catch, filling the existing `ScopedCapabilityResult.error` seam instead + of rejecting the pool. +- **Bounded observation.** `renderObservation` clips text and ARIA blocks at + 8,000 chars each with an explicit `[truncated N chars]` marker; console and + network windows were already bounded. +- **Visible cost and conduct.** `ModelResponse.usage` (adapter-mapped from + both providers) accumulates into `DriveResult.tokens`, rendered in the QA + summary. `DriveOptions.onStep` emits a per-turn audit event (tool calls, + outcomes, model latency); the CLI's `--verbose` writes it to stderr as it + happens. + +## Constraints + +- No verdict semantics change: stable still means N green attempts; an errored + attempt is simply a failed one with a reason. +- Additive public surface only (`usage`, `tokens`, `errors`, `onStep`, + timeout options); every existing caller compiles unchanged. +- The emitted spec's helper must match `runCommand` byte-for-byte in + behavior — the two are changed together. + +## Rationale + +Timeouts belong at each shell-out/await site (the only places a hang can +start), not in a global watchdog that would kill work it cannot attribute. +Retrying exactly once catches the dominant transient-blip case without hiding +a dead provider. Filling the pool's existing `error` field keeps the scoped +result shape stable for the PR-comment renderer. + +## Alternatives + +- **AbortController threaded through ModelClient.** Deferred: it changes the + BYO-model interface every custom adapter implements; a race-based timeout + unblocks the loop today and a signal can be added additively later. +- **A global drive watchdog.** Rejected: coarser than per-site caps and it + cannot say *what* hung. +- **Configurable observation budget.** Deferred until a real page needs it; + the marker makes truncation visible when it happens. + +## Accessibility + +Not applicable — timeouts and logging; the `--verbose` stream is plain text. + +## Style Guidance + +Timeout errors name the cap that fired ("model call timed out after 120000ms") +and retry errors name both failures, so a transcript reads as a diagnosis. + +## Open Questions + +- Whether a spend ceiling (`--max-tokens-budget`) should abort a drive + mid-run. Usage is now measured, which is the prerequisite. + +## Related Requirements + +- req-drive-resilience + +## Related Roadmaps + +- autonomous-qa-enhancements + +## Status + +Accepted diff --git a/lore-proofkeeper/requirements/req-drive-resilience.md b/lore-proofkeeper/requirements/req-drive-resilience.md new file mode 100644 index 0000000..6c695f4 --- /dev/null +++ b/lore-proofkeeper/requirements/req-drive-resilience.md @@ -0,0 +1,62 @@ +--- +schema_version: 1 +id: PK-KWFWJMV2DBWA +type: requirement +--- +# Drive Resilience — Timeouts, Isolation, Budgets, Audit + +## Problem + +Nothing in the pipeline bounded wall-clock time or spend, and one failure could +destroy unrelated work. A stalled model call, a hung shell command, a hung +Playwright or rac process each blocked the pipeline forever; a single transient +provider error aborted an otherwise-recoverable capability; a throwing drive +inside a scoped run rejected the pool and discarded every sibling capability's +completed result; a runner exception aborted the fidelity gate instead of +counting as a failed attempt. Meanwhile the transcript grew unbounded (a full +observation re-appended every turn), token spend was invisible, and the loop +left no record of what the agent actually did. + +## Requirements + +- [REQ-001] Every external await is time-boxed: model calls, shell commands (recorded and replayed), Playwright invocations, the rac graph export, and the git diff each carry a wall-clock cap whose expiry surfaces as an error, never a hang. +- [REQ-002] A failed model call is retried once with backoff before the drive gives up, and the final error names both failures. +- [REQ-003] A runner exception during the fidelity gate counts as a failed attempt with a recorded reason on the verdict; the gate always completes its N attempts. +- [REQ-004] In a scoped run, one capability's exception becomes that capability's error entry; sibling capabilities' results are never discarded. +- [REQ-005] Observation text and ARIA blocks are capped per turn with an explicit truncation marker, bounding transcript growth. +- [REQ-006] Provider-reported token usage is surfaced by the bundled adapters, accumulated per drive, and shown in the QA summary; `--verbose` logs each turn's tool calls, errors, and model latency as an audit trail. + +## Success Metrics + +- A stalled model call errors at the cap instead of hanging; a transient 5xx + no longer aborts a capability. +- A scoped run with one throwing capability still reports every sibling's + verdict. +- A drive on a usage-reporting provider prints its token totals; a hung + `sleep`-style command errors within its cap. + +## Risks + +- Caps that are too tight fail slow-but-healthy runs. Mitigation: generous + defaults (2 min model/command, 10 min per Playwright invocation), and the + model timeout and command timeout are overridable. +- A retry doubles cost on genuinely dead providers. Mitigation: exactly one + retry, with both errors reported. + +## Assumptions + +- Provider `usage` fields (Anthropic `input_tokens`/`output_tokens`, OpenAI + `prompt_tokens`/`completion_tokens`) remain stable contract surfaces. +- The head of a page's text/ARIA carries the signal locators need, so clipping + the tail loses little. + +## Related Roadmaps + +- autonomous-qa-enhancements + +## Verified By + +- `tests/drive-loop.test.ts` +- `tests/fidelity.test.ts` +- `tests/scoped-qa.test.ts` +- `tests/observe.test.ts` diff --git a/lore-proofkeeper/roadmaps/autonomous-qa-enhancements.md b/lore-proofkeeper/roadmaps/autonomous-qa-enhancements.md index 76db5c8..5ce9587 100644 --- a/lore-proofkeeper/roadmaps/autonomous-qa-enhancements.md +++ b/lore-proofkeeper/roadmaps/autonomous-qa-enhancements.md @@ -86,6 +86,14 @@ finish, assertion-free sessions refused, exact locator matching on record and replay, and contract anomalies (schema versions, empty or retried reports) refused rather than guessed at. Serves the faithful-tests outcome at its core. +### Drive resilience + +Bound every external await (model, shell, Playwright, rac, git) with a +wall-clock cap, retry transient model failures once, isolate one capability's +failure from its siblings, cap observation growth, and make token spend and +per-turn conduct visible. Serves the fast-reliable-scoped-QA outcome under +real-world failure. + ## Success Measures - A pull request shows exactly one Proofkeeper QA comment regardless of how many diff --git a/src/agent/adapters/claude.ts b/src/agent/adapters/claude.ts index ccbc9e1..face72e 100644 --- a/src/agent/adapters/claude.ts +++ b/src/agent/adapters/claude.ts @@ -38,6 +38,7 @@ interface AnthropicContentBlock { interface AnthropicMessage { stop_reason?: string; content: AnthropicContentBlock[]; + usage?: { input_tokens?: number; output_tokens?: number }; } /** The slice of the Anthropic SDK client this adapter calls. */ @@ -100,8 +101,12 @@ export function fromAnthropicResponse(message: AnthropicMessage): ModelResponse textParts.push(block.text); } } - if (toolCalls.length > 0) return { toolCalls }; - return { done: textParts.join("\n") }; + const usage = + message.usage !== undefined + ? { usage: { inputTokens: message.usage.input_tokens ?? 0, outputTokens: message.usage.output_tokens ?? 0 } } + : {}; + if (toolCalls.length > 0) return { toolCalls, ...usage }; + return { done: textParts.join("\n"), ...usage }; } export class ClaudeModelClient implements ModelClient { diff --git a/src/agent/adapters/openai.ts b/src/agent/adapters/openai.ts index 59e7775..ac2cda8 100644 --- a/src/agent/adapters/openai.ts +++ b/src/agent/adapters/openai.ts @@ -54,6 +54,7 @@ interface OpenAIResponseMessage { interface OpenAICompletion { choices?: { message?: OpenAIResponseMessage }[]; + usage?: { prompt_tokens?: number; completion_tokens?: number }; } /** The slice of `fetch` this adapter calls. Inject a double for tests. */ @@ -116,8 +117,17 @@ export function fromOpenAIResponse(completion: OpenAICompletion): ModelResponse const name = call.function?.name; if (name) toolCalls.push({ name, arguments: parseToolArguments(call.function?.arguments) }); } - if (toolCalls.length > 0) return { toolCalls }; - return { done: message.content ?? "" }; + const usage = + completion.usage !== undefined + ? { + usage: { + inputTokens: completion.usage.prompt_tokens ?? 0, + outputTokens: completion.usage.completion_tokens ?? 0, + }, + } + : {}; + if (toolCalls.length > 0) return { toolCalls, ...usage }; + return { done: message.content ?? "", ...usage }; } export class OpenAICompatibleModelClient implements ModelClient { diff --git a/src/agent/drive.ts b/src/agent/drive.ts index e102775..78bfeb9 100644 --- a/src/agent/drive.ts +++ b/src/agent/drive.ts @@ -18,7 +18,7 @@ import type { Page } from "@playwright/test"; import type { RecordedSession } from "../compiler/actions.js"; import { Recorder } from "../compiler/recorder.js"; import { observePage, renderObservation, createPageMonitor } from "./observe.js"; -import type { ModelClient, ModelRequest, ToolCall } from "./model.js"; +import type { ModelClient, ModelRequest, ModelResponse, ToolCall } from "./model.js"; import { toolsForPolicy, LOCATOR_GUIDANCE, @@ -37,6 +37,9 @@ import { redactText } from "./redact.js"; const DEFAULT_MAX_STEPS = 12; +/** Default wall-clock cap on one model call. */ +export const DEFAULT_MODEL_TIMEOUT_MS = 120_000; + export interface DriveOptions { /** Capability under verification; threads into the recorded session. */ capabilityId?: string; @@ -73,6 +76,29 @@ export interface DriveOptions { * origin (`--allow-host` / config `allowedHosts`). Everything else is refused. */ allowedHosts?: string[]; + /** + * Wall-clock cap on one model call. A stalled provider must not hang the + * drive forever. Defaults to {@link DEFAULT_MODEL_TIMEOUT_MS}. + */ + modelTimeoutMs?: number; + /** + * Backoff before the single retry of a failed model call. Defaults to 2s; + * tests pass 0. + */ + modelRetryBackoffMs?: number; + /** Per-turn observer for the audit trail (`--verbose`): what the agent did. */ + onStep?: (event: DriveStepEvent) => void; +} + +/** One turn's audit record, emitted through {@link DriveOptions.onStep}. */ +export interface DriveStepEvent { + step: number; + /** Tool names the model called this turn (empty on a give-up turn). */ + calls: string[]; + /** Per-call outcome lines (`ok: click`, `ERROR navigate: …`). */ + outcomes: string[]; + /** Model-call latency for the turn, in milliseconds. */ + modelMs: number; } export interface DriveResult { @@ -80,6 +106,8 @@ export interface DriveResult { session: RecordedSession; /** True only when the model explicitly called `finish`. */ finished: boolean; + /** Accumulated provider-reported token usage, when the adapter surfaces it. */ + tokens?: { input: number; output: number }; /** * Why the drive ended: `finished` — the model called `finish`; `gave_up` — * the model stopped issuing tool calls without finishing; `step_budget` — @@ -239,6 +267,28 @@ async function dispatch(recorder: Recorder, call: ToolCall, policy: EgressPolicy } } +/** Reject when the model call outlives the cap — a stalled provider must not hang the drive. */ +function withTimeout(promise: Promise, ms: number): Promise { + return new Promise((resolve, reject) => { + const timer = setTimeout( + () => reject(new Error(`model call timed out after ${ms}ms`)), + ms, + ); + promise.then( + (value) => { + clearTimeout(timer); + resolve(value); + }, + (err: unknown) => { + clearTimeout(timer); + reject(err instanceof Error ? err : new Error(String(err))); + }, + ); + }); +} + +const sleep = (ms: number): Promise => new Promise((r) => setTimeout(r, ms)); + export class AutonomousDriver { constructor( private readonly page: Page, @@ -246,6 +296,28 @@ export class AutonomousDriver { private readonly options: DriveOptions, ) {} + /** + * One model call with a wall-clock cap and a single backed-off retry — a + * transient provider blip must not abort an otherwise-recoverable drive. + * (Action errors already feed back to the model; this covers the model call + * itself, which previously had no timeout or retry at all.) + */ + private async complete(request: ModelRequest): Promise { + const timeoutMs = this.options.modelTimeoutMs ?? DEFAULT_MODEL_TIMEOUT_MS; + try { + return await withTimeout(this.model.complete(request), timeoutMs); + } catch (first) { + await sleep(this.options.modelRetryBackoffMs ?? 2000); + try { + return await withTimeout(this.model.complete(request), timeoutMs); + } catch (second) { + throw new Error( + `model call failed twice: ${(first as Error).message}; retry: ${(second as Error).message}`, + ); + } + } + } + async drive(): Promise { // The trust boundary for this drive: shell off unless opted in; egress // limited to the start URL's origin, the extension's pages, and any @@ -291,11 +363,22 @@ export class AutonomousDriver { // Optional planning turn: ask for a Markdown test plan (no tools → text), // record it, and feed it back as context so the drive follows its own plan. let plan: string | undefined; + const tokens = { input: 0, output: 0 }; + let sawUsage = false; + const account = (response: ModelResponse): void => { + if (response.usage) { + sawUsage = true; + tokens.input += response.usage.inputTokens; + tokens.output += response.usage.outputTokens; + } + }; + if (this.options.plan) { - const response = await this.model.complete({ + const response = await this.complete({ transcript: [...transcript, { role: "user", content: PLAN_INSTRUCTION }], tools: [], }); + account(response); const text = response.done?.trim(); if (text) { plan = text; @@ -311,7 +394,10 @@ export class AutonomousDriver { while (steps < maxSteps) { steps++; - const response = await this.model.complete({ transcript, tools: toolsForPolicy(policy) }); + const modelStart = Date.now(); + const response = await this.complete({ transcript, tools: toolsForPolicy(policy) }); + const modelMs = Date.now() - modelStart; + account(response); const calls = response.toolCalls ?? []; if (calls.length === 0) { @@ -320,6 +406,7 @@ export class AutonomousDriver { // there are no tool calls, so its mere presence proves nothing.) stopReason = "gave_up"; gaveUpText = response.done?.trim() || undefined; + this.options.onStep?.({ step: steps, calls: [], outcomes: ["(no tool calls — gave up)"], modelMs }); break; } @@ -332,6 +419,7 @@ export class AutonomousDriver { if (result.finished) { finished = true; stopReason = "finished"; + outcomes.push("finish"); stop = true; break; } @@ -341,6 +429,7 @@ export class AutonomousDriver { : `ERROR ${call.name}: ${result.error}`, ); } + this.options.onStep?.({ step: steps, calls: calls.map((c) => c.name), outcomes, modelMs }); if (stop) break; transcript.push({ role: "user", content: `Results:\n${outcomes.join("\n")}\n\n${await observe()}` }); @@ -354,6 +443,7 @@ export class AutonomousDriver { finished, stopReason, ...(gaveUpText !== undefined ? { gaveUpText } : {}), + ...(sawUsage ? { tokens } : {}), steps, ...(plan !== undefined ? { plan } : {}), }; diff --git a/src/agent/model.ts b/src/agent/model.ts index c8ed563..90a09dd 100644 --- a/src/agent/model.ts +++ b/src/agent/model.ts @@ -31,6 +31,11 @@ export interface ModelResponse { toolCalls?: ToolCall[]; /** Terminal assistant message when the model decides the session is done. */ done?: string; + /** + * Provider-reported token usage for this turn, when the adapter surfaces it. + * The drive accumulates it so a run's cost is visible, not invisible. + */ + usage?: { inputTokens: number; outputTokens: number }; } /** A caller-supplied model. Proofkeeper bundles none. */ diff --git a/src/agent/observe.ts b/src/agent/observe.ts index cdded7b..b88c662 100644 --- a/src/agent/observe.ts +++ b/src/agent/observe.ts @@ -40,13 +40,26 @@ export async function observePage(page: Page): Promise { return { url: page.url(), title, text: text.trim(), aria: aria.trim() }; } +/** + * Per-block character budget for an observation. A full observation is + * re-appended to the transcript every turn, so an unbounded content-heavy page + * grows the prompt quadratically over a drive; the head of the text and ARIA + * tree carries the signal locators need. + */ +export const OBSERVATION_BLOCK_BUDGET = 8_000; + +function clip(block: string, budget = OBSERVATION_BLOCK_BUDGET): string { + if (block.length <= budget) return block; + return `${block.slice(0, budget)}\n… [truncated ${block.length - budget} chars]`; +} + /** Render an observation as the text block fed to the model. */ export function renderObservation(o: PageObservation): string { const blocks = [ `URL: ${o.url}`, `Title: ${o.title}`, - `Visible text:\n${o.text}`, - `Accessibility tree:\n${o.aria}`, + `Visible text:\n${clip(o.text)}`, + `Accessibility tree:\n${clip(o.aria)}`, ]; if (o.console && o.console.length > 0) blocks.push(`Console:\n${o.console.join("\n")}`); if (o.network && o.network.length > 0) blocks.push(`Network:\n${o.network.join("\n")}`); diff --git a/src/cli.ts b/src/cli.ts index 442231f..1140b7a 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -90,6 +90,8 @@ qa options: explicit operator decision). --allow-host Allow navigate/request to this hostname in addition to the start URL's origin (repeatable). All other egress is refused. + --verbose Log each drive turn (tool calls, errors, model latency) to + stderr as it happens — the audit trail of what the agent did. --propose Propose a Verified By write-back PR when the test is stable. --target-path Artifact to write back to (required with --propose). --repo Target repository for the write-back (required with --propose). @@ -280,6 +282,7 @@ export interface QaArgs { extensionPath?: string; allowShell: boolean; allowedHosts: string[]; + verbose: boolean; propose: boolean; targetPath?: string; base?: string; @@ -345,6 +348,9 @@ export function parseQaArgs(argv: string[]): QaArgs { case "--allow-host": (raw.allowedHosts ??= []).push(requireValue(argv[++i], "--allow-host")); break; + case "--verbose": + raw.verbose = true; + break; case "--propose": raw.propose = true; break; @@ -392,6 +398,7 @@ export function parseQaArgs(argv: string[]): QaArgs { ...(raw.extensionPath !== undefined ? { extensionPath: raw.extensionPath } : {}), allowShell: raw.allowShell ?? false, allowedHosts: raw.allowedHosts ?? [], + verbose: raw.verbose ?? false, propose: raw.propose ?? false, ...(raw.targetPath !== undefined ? { targetPath: raw.targetPath } : {}), ...(raw.base !== undefined ? { base: raw.base } : {}), @@ -424,8 +431,28 @@ function resolveModel(): ModelClient { } /** A browser-backed drive seam: launch Chromium, drive, always close. */ -function browserDrive(model: ModelClient): (options: DriveOptions) => Promise { - return async (options) => { +function browserDrive( + model: ModelClient, + seamOptions: { verbose?: boolean } = {}, +): (options: DriveOptions) => Promise { + // The audit trail: with --verbose, every turn's tool calls and outcomes are + // logged to stderr as they happen — an autonomous agent should leave a + // record of what it did, not only a final summary. + const onStep = + seamOptions.verbose === true + ? (event: import("./agent/drive.js").DriveStepEvent): void => { + const summary = event.calls.length > 0 ? event.calls.join(", ") : "(no tool calls)"; + const errors = event.outcomes.filter((o) => o.startsWith("ERROR")); + process.stderr.write( + ` step ${event.step} (model ${(event.modelMs / 1000).toFixed(1)}s): ${summary}` + + (errors.length > 0 ? `\n ${errors.join("\n ")}` : "") + + "\n", + ); + } + : undefined; + + return async (rawOptions) => { + const options: DriveOptions = { ...rawOptions, ...(onStep ? { onStep } : {}) }; const { chromium } = await import("@playwright/test"); // Extension verification needs a persistent context with the unpacked @@ -504,7 +531,7 @@ async function runQaCommand(argv: string[]): Promise { : {}), }; const deps: QaDeps = { - drive: browserDrive(model), + drive: browserDrive(model, { verbose: args.verbose }), compiler: new CodegenCompiler({ outDir: args.outDir }), runner: new PlaywrightRunner(), learning: new FileLearningStore(), @@ -527,12 +554,16 @@ function renderQaResult(result: Awaited>): string { `Capability: ${result.capability.id} — ${result.capability.title}`, `Drive: ${result.drive.steps} step(s), ${stop}`, ]; + if (result.drive.tokens) { + lines.push(`Tokens: ${result.drive.tokens.input} in / ${result.drive.tokens.output} out`); + } if (result.loop) { const v = result.loop.verdict; lines.push( `Compiled: ${result.loop.candidate.specPath}`, `Fidelity: ${v.passed}/${v.attempts} re-runs green — ${v.stable ? "stable" : "unstable, quarantined"}`, ); + for (const error of v.errors ?? []) lines.push(` runner error — ${error}`); } else { lines.push(`Not compiled: ${result.unverifiedReason ?? "nothing to verify"}`); } @@ -666,7 +697,10 @@ export function parseScopedArgs(argv: string[]): ScopedArgs { /** Files changed against a git ref, as `git diff --name-only `. */ async function gitChangedFiles(baseRef: string): Promise { try { - const { stdout } = await execFileAsync("git", ["diff", "--name-only", baseRef], { maxBuffer: 16 * 1024 * 1024 }); + const { stdout } = await execFileAsync("git", ["diff", "--name-only", baseRef], { + maxBuffer: 16 * 1024 * 1024, + timeout: 60_000, + }); return stdout.split("\n").map((s) => s.trim()).filter(Boolean); } catch (err) { throw new UsageError(`git diff --name-only ${baseRef} failed: ${(err as Error).message}`); diff --git a/src/compiler/emit.ts b/src/compiler/emit.ts index a6c9e9a..7f7096c 100644 --- a/src/compiler/emit.ts +++ b/src/compiler/emit.ts @@ -172,7 +172,7 @@ export function emitSpec(session: RecordedSession): string { // helper exactly, so a recording that held re-runs green. const terminalHelper = terminal ? `\nfunction runCommand(command: string, options: { cwd?: string } = {}) { - const r = spawnSync(command, { shell: true, encoding: "utf8", ...options }); + const r = spawnSync(command, { shell: true, encoding: "utf8", timeout: 120000, maxBuffer: 16777216, ...options }); if (r.error) throw r.error; return { stdout: r.stdout ?? "", stderr: r.stderr ?? "", code: r.status ?? 0 }; }\n` diff --git a/src/compiler/terminal.ts b/src/compiler/terminal.ts index bc6fb8c..2e8d305 100644 --- a/src/compiler/terminal.ts +++ b/src/compiler/terminal.ts @@ -26,6 +26,16 @@ export interface OutputAssertion { value: string; } +/** + * Wall-clock cap on one command. A hung command (a server that never exits, a + * prompt waiting for input) must not hang the whole drive; the timeout error + * surfaces to the model as a failed action. Mirrored in the emitted spec. + */ +export const COMMAND_TIMEOUT_MS = 120_000; + +/** Output cap per stream — beyond this the command errors instead of ENOBUFS. */ +export const COMMAND_MAX_BUFFER = 16 * 1024 * 1024; + /** * Run a shell command and capture its result. Uses `shell: true` so a recorded * command string (pipes, args, redirects) runs as written. The command executes @@ -33,12 +43,18 @@ export interface OutputAssertion { * committed test will when re-run (the trust boundary stays human PR review, * ADR-065). * - * @throws when the process could not be spawned (a runner error, not a verdict). + * @throws when the process could not be spawned, timed out, or overflowed the + * output cap (a runner error, not a verdict). */ -export function runCommand(command: string, options: { cwd?: string } = {}): CommandResult { +export function runCommand( + command: string, + options: { cwd?: string; timeoutMs?: number } = {}, +): CommandResult { const result = spawnSync(command, { shell: true, encoding: "utf8", + timeout: options.timeoutMs ?? COMMAND_TIMEOUT_MS, + maxBuffer: COMMAND_MAX_BUFFER, ...(options.cwd !== undefined ? { cwd: options.cwd } : {}), }); if (result.error) throw result.error; diff --git a/src/coverage/source.ts b/src/coverage/source.ts index 9a04ad2..9052e9d 100644 --- a/src/coverage/source.ts +++ b/src/coverage/source.ts @@ -41,6 +41,8 @@ export async function loadGraphFromCorpus(corpusDir: string, racBin = "rac"): Pr try { ({ stdout } = await execFileAsync(racBin, ["export", corpusDir, "--graph"], { maxBuffer: 64 * 1024 * 1024, + // A hung or interactive rac must not hang the pipeline. + timeout: 120_000, })); } catch (err) { throw new GraphParseError( diff --git a/src/fidelity/gate.ts b/src/fidelity/gate.ts index af8cd92..dbd5a7b 100644 --- a/src/fidelity/gate.ts +++ b/src/fidelity/gate.ts @@ -27,6 +27,12 @@ export interface FidelityVerdict { passed: number; /** Per-attempt pass/fail, in order. */ runs: boolean[]; + /** + * Runner errors that counted as failed attempts (infrastructure failures — + * a hung browser, a missing report). Present only when at least one attempt + * errored, so a reviewer can tell "the test failed" from "the run broke". + */ + errors?: string[]; } /** @@ -44,14 +50,24 @@ export async function assessFidelity( } const runs: boolean[] = []; + const errors: string[] = []; for (let attempt = 0; attempt < options.n; attempt++) { - const results = await runner.run([test], { - targets: [options.target], - parallelism: options.parallelism, - }); - // One test, one target ⇒ a single result. Treat a missing result as failure. - const passed = results.length > 0 && results.every((r) => r.status === "passed"); - runs.push(passed); + try { + const results = await runner.run([test], { + targets: [options.target], + parallelism: options.parallelism, + }); + // One test, one target ⇒ a single result. Treat a missing result as failure. + const passed = results.length > 0 && results.every((r) => r.status === "passed"); + runs.push(passed); + } catch (err) { + // A runner error (hung browser, missing report, broken install) is a + // failed attempt with a recorded reason — quarantine, not an abort that + // would discard the other attempts (and, in scoped runs, sibling + // capabilities). + runs.push(false); + errors.push(`attempt ${attempt + 1}: ${(err as Error).message}`); + } } const passed = runs.filter(Boolean).length; @@ -61,5 +77,6 @@ export async function assessFidelity( attempts: options.n, passed, runs, + ...(errors.length > 0 ? { errors } : {}), }; } diff --git a/src/index.ts b/src/index.ts index 73b0116..f71bbf5 100644 --- a/src/index.ts +++ b/src/index.ts @@ -80,8 +80,8 @@ export type { ScaffoldOptions } from "./scaffold/scaffold.js"; export { runAgentLoop } from "./agent/loop.js"; export type { AgentLoopDeps, AgentLoopOptions, AgentLoopResult } from "./agent/loop.js"; export type { ModelClient, ModelRequest, ModelResponse, ToolCall } from "./agent/model.js"; -export { AutonomousDriver, runDrive } from "./agent/drive.js"; -export type { DriveOptions, DriveResult } from "./agent/drive.js"; +export { AutonomousDriver, runDrive, DEFAULT_MODEL_TIMEOUT_MS } from "./agent/drive.js"; +export type { DriveOptions, DriveResult, DriveStepEvent } from "./agent/drive.js"; export { DRIVE_TOOLS, toolsForPolicy, diff --git a/src/qa/run-scoped.ts b/src/qa/run-scoped.ts index f6845dc..bef9d7e 100644 --- a/src/qa/run-scoped.ts +++ b/src/qa/run-scoped.ts @@ -149,24 +149,31 @@ export async function runScopedQa(deps: ScopedQaDeps, options: ScopedQaOptions): ...(deps.learning ? { learning: deps.learning } : {}), }; - const result = await runQa(capDeps, { - graph: options.graph, - capabilityId: cap.id, - startUrl: target.url, - ...(cap.config.goal !== undefined ? { goal: cap.config.goal } : {}), - ...(goalContext !== undefined ? { goalContext } : {}), - // Each capability runs against its resolved environment URL. - target: { name: target.name, baseURL: target.url }, - n: options.n, - ...(options.maxSteps !== undefined ? { maxSteps: options.maxSteps } : {}), - ...(options.plan ? { plan: true } : {}), - ...(target.extensionPath !== undefined ? { extensionPath: target.extensionPath } : {}), - // The config's trust boundary: shell opt-in and host allowlist. - ...(options.config.allowShell !== undefined ? { allowShell: options.config.allowShell } : {}), - ...(options.config.allowedHosts !== undefined ? { allowedHosts: options.config.allowedHosts } : {}), - ...(propose ? { propose } : {}), - }); - return { capability: cap, result }; + // Isolate the whole drive→gate→propose per capability: one capability's + // throw (a model outage, a broken runner) becomes ITS error entry — it + // must never reject the pool and discard sibling capabilities' results. + try { + const result = await runQa(capDeps, { + graph: options.graph, + capabilityId: cap.id, + startUrl: target.url, + ...(cap.config.goal !== undefined ? { goal: cap.config.goal } : {}), + ...(goalContext !== undefined ? { goalContext } : {}), + // Each capability runs against its resolved environment URL. + target: { name: target.name, baseURL: target.url }, + n: options.n, + ...(options.maxSteps !== undefined ? { maxSteps: options.maxSteps } : {}), + ...(options.plan ? { plan: true } : {}), + ...(target.extensionPath !== undefined ? { extensionPath: target.extensionPath } : {}), + // The config's trust boundary: shell opt-in and host allowlist. + ...(options.config.allowShell !== undefined ? { allowShell: options.config.allowShell } : {}), + ...(options.config.allowedHosts !== undefined ? { allowedHosts: options.config.allowedHosts } : {}), + ...(propose ? { propose } : {}), + }); + return { capability: cap, result }; + } catch (err) { + return { capability: cap, error: (err as Error).message }; + } }, ); diff --git a/src/runner/playwright-runner.ts b/src/runner/playwright-runner.ts index d39aac3..44f1702 100644 --- a/src/runner/playwright-runner.ts +++ b/src/runner/playwright-runner.ts @@ -26,6 +26,9 @@ interface ExecError extends Error { stderr?: string; } +/** Wall-clock cap on one Playwright invocation — a hung browser must not hang the pipeline. */ +export const RUN_TIMEOUT_MS = 10 * 60_000; + export interface PlaywrightRunnerOptions { /** Working directory the Playwright project lives in. Defaults to cwd. */ cwd?: string; @@ -37,17 +40,21 @@ export interface PlaywrightRunnerOptions { outputDir?: string; /** Override the Playwright invocation (advanced/testing). */ command?: { bin: string; baseArgs: string[] }; + /** Wall-clock cap per Playwright invocation. Defaults to {@link RUN_TIMEOUT_MS}. */ + timeoutMs?: number; } export class PlaywrightRunner implements Runner { private readonly cwd: string; private readonly outputDir: string | undefined; private readonly command: { bin: string; baseArgs: string[] }; + private readonly timeoutMs: number; constructor(options: PlaywrightRunnerOptions = {}) { this.cwd = options.cwd ?? process.cwd(); this.outputDir = options.outputDir; this.command = options.command ?? { bin: "npx", baseArgs: ["playwright", "test"] }; + this.timeoutMs = options.timeoutMs ?? RUN_TIMEOUT_MS; } async run(suite: CompiledTest[], options: RunOptions): Promise { @@ -85,6 +92,7 @@ export class PlaywrightRunner implements Runner { cwd: this.cwd, env, maxBuffer: 256 * 1024 * 1024, + timeout: this.timeoutMs, })); } catch (err) { const execErr = err as ExecError; diff --git a/tests/claude-adapter.test.ts b/tests/claude-adapter.test.ts index ad6a13d..a950d4b 100644 --- a/tests/claude-adapter.test.ts +++ b/tests/claude-adapter.test.ts @@ -73,6 +73,23 @@ describe("fromAnthropicResponse", () => { expect(result.toolCalls).toBeUndefined(); }); + it("surfaces provider-reported usage on both tool-call and done turns", () => { + const withTools = fromAnthropicResponse({ + content: [{ type: "tool_use", name: "finish", input: {} }], + usage: { input_tokens: 120, output_tokens: 15 }, + }); + expect(withTools.usage).toEqual({ inputTokens: 120, outputTokens: 15 }); + + const doneTurn = fromAnthropicResponse({ + content: [{ type: "text", text: "done" }], + usage: { input_tokens: 80, output_tokens: 5 }, + }); + expect(doneTurn.usage).toEqual({ inputTokens: 80, outputTokens: 5 }); + + const noUsage = fromAnthropicResponse({ content: [{ type: "text", text: "done" }] }); + expect(noUsage.usage).toBeUndefined(); + }); + it("defaults missing tool input to an empty object", () => { const result = fromAnthropicResponse({ content: [{ type: "tool_use", name: "finish" }] }); expect(result.toolCalls).toEqual([{ name: "finish", arguments: {} }]); diff --git a/tests/drive-loop.test.ts b/tests/drive-loop.test.ts index 9cba934..035edaa 100644 --- a/tests/drive-loop.test.ts +++ b/tests/drive-loop.test.ts @@ -125,3 +125,77 @@ describe("drive trust boundary in the loop", () => { expect(result.session.actions).toEqual([{ type: "goto", url: "http://x/" }]); }); }); + +describe("drive resilience", () => { + it("retries a transient model failure once and continues", async () => { + let calls = 0; + const flaky: ModelClient = { + complete: (): Promise => { + calls++; + if (calls === 1) return Promise.reject(new Error("502 upstream")); + return Promise.resolve({ toolCalls: [{ name: "finish", arguments: {} }] }); + }, + }; + const result = await new AutonomousDriver(fakePage(), flaky, { + ...OPTIONS, + modelRetryBackoffMs: 0, + }).drive(); + + expect(result.finished).toBe(true); + expect(calls).toBe(2); + }); + + it("fails with both errors when the model call fails twice", async () => { + const dead: ModelClient = { complete: () => Promise.reject(new Error("502 upstream")) }; + await expect( + new AutonomousDriver(fakePage(), dead, { ...OPTIONS, modelRetryBackoffMs: 0 }).drive(), + ).rejects.toThrow(/model call failed twice: 502 upstream; retry: 502 upstream/); + }); + + it("times out a stalled model call instead of hanging the drive", async () => { + const stalled: ModelClient = { complete: () => new Promise(() => undefined) }; + await expect( + new AutonomousDriver(fakePage(), stalled, { + ...OPTIONS, + modelTimeoutMs: 20, + modelRetryBackoffMs: 0, + }).drive(), + ).rejects.toThrow(/timed out after 20ms/); + }); + + it("accumulates provider-reported token usage across turns", async () => { + const model = new ScriptedModel([ + { + toolCalls: [{ name: "navigate", arguments: { url: "http://x/a" } }], + usage: { inputTokens: 100, outputTokens: 10 }, + }, + { toolCalls: [{ name: "finish", arguments: {} }], usage: { inputTokens: 200, outputTokens: 20 } }, + ]); + const result = await new AutonomousDriver(fakePage(), model, OPTIONS).drive(); + + expect(result.tokens).toEqual({ input: 300, output: 30 }); + }); + + it("reports no tokens when the model surfaces no usage", async () => { + const model = new ScriptedModel([{ toolCalls: [{ name: "finish", arguments: {} }] }]); + const result = await new AutonomousDriver(fakePage(), model, OPTIONS).drive(); + expect(result.tokens).toBeUndefined(); + }); + + it("emits a per-turn audit event through onStep", async () => { + const events: { step: number; calls: string[]; outcomes: string[] }[] = []; + const model = new ScriptedModel([ + { toolCalls: [{ name: "navigate", arguments: { url: "https://evil.example.net/" } }] }, + { toolCalls: [{ name: "finish", arguments: {} }] }, + ]); + await new AutonomousDriver(fakePage(), model, { + ...OPTIONS, + onStep: (e) => events.push({ step: e.step, calls: e.calls, outcomes: e.outcomes }), + }).drive(); + + expect(events.map((e) => e.step)).toEqual([1, 2]); + expect(events[0]?.calls).toEqual(["navigate"]); + expect(events[0]?.outcomes.join()).toContain("ERROR navigate"); + expect(events[1]?.outcomes).toEqual(["finish"]); + }); +}); diff --git a/tests/fidelity.test.ts b/tests/fidelity.test.ts index 338d458..3f88ef4 100644 --- a/tests/fidelity.test.ts +++ b/tests/fidelity.test.ts @@ -70,3 +70,29 @@ describe("assessFidelity", () => { ); }); }); + +describe("assessFidelity — runner errors are verdicts, not aborts", () => { + class ThrowOnceRunner implements Runner { + private call = 0; + run(suite: CompiledTest[], _options: RunOptions): Promise { + this.call++; + if (this.call === 2) return Promise.reject(new Error("playwright run failed: browser hung")); + return Promise.resolve( + suite.map((t) => ({ testId: t.id, target: TARGET.name, status: "passed" as const, durationMs: 1 })), + ); + } + } + + it("counts a runner exception as a failed attempt with a recorded reason", async () => { + const verdict = await assessFidelity(new ThrowOnceRunner(), TEST, { n: 3, target: TARGET }); + expect(verdict.stable).toBe(false); + expect(verdict.passed).toBe(2); + expect(verdict.runs).toEqual([true, false, true]); + expect(verdict.errors).toEqual(["attempt 2: playwright run failed: browser hung"]); + }); + + it("omits errors entirely when no attempt errored", async () => { + const verdict = await assessFidelity(new ScriptedRunner([true, true]), TEST, { n: 2, target: TARGET }); + expect(verdict.errors).toBeUndefined(); + }); +}); diff --git a/tests/observe.test.ts b/tests/observe.test.ts index baaa819..005de6b 100644 --- a/tests/observe.test.ts +++ b/tests/observe.test.ts @@ -73,3 +73,23 @@ describe("createPageMonitor", () => { expect(monitor.console).toEqual([]); }); }); + +describe("observation budget", () => { + it("clips oversized text and ARIA blocks with a truncation marker", () => { + const rendered = renderObservation({ + url: "http://x/", + title: "t", + text: "a".repeat(9000), + aria: "b".repeat(8100), + }); + expect(rendered).toContain("… [truncated 1000 chars]"); + expect(rendered).toContain("… [truncated 100 chars]"); + // Bounded: nowhere near the raw 17k of input. + expect(rendered.length).toBeLessThan(17000); + }); + + it("leaves small observations untouched", () => { + const rendered = renderObservation({ url: "http://x/", title: "t", text: "hello", aria: "- doc" }); + expect(rendered).not.toContain("truncated"); + }); +}); diff --git a/tests/openai-adapter.test.ts b/tests/openai-adapter.test.ts index 7e6c915..f01a7c0 100644 --- a/tests/openai-adapter.test.ts +++ b/tests/openai-adapter.test.ts @@ -93,6 +93,23 @@ describe("fromOpenAIResponse", () => { ).toEqual([{ name: "finish", arguments: {} }]); }); + it("surfaces provider-reported usage on both tool-call and done turns", () => { + const withTools = fromOpenAIResponse({ + choices: [{ message: { tool_calls: [{ function: { name: "finish", arguments: "{}" } }] } }], + usage: { prompt_tokens: 120, completion_tokens: 15 }, + }); + expect(withTools.usage).toEqual({ inputTokens: 120, outputTokens: 15 }); + + const doneTurn = fromOpenAIResponse({ + choices: [{ message: { content: "done" } }], + usage: { prompt_tokens: 80, completion_tokens: 5 }, + }); + expect(doneTurn.usage).toEqual({ inputTokens: 80, outputTokens: 5 }); + + const noUsage = fromOpenAIResponse({ choices: [{ message: { content: "done" } }] }); + expect(noUsage.usage).toBeUndefined(); + }); + it("returns an empty done when the provider sends no content and no tool calls", () => { expect(fromOpenAIResponse({ choices: [{ message: {} }] }).done).toBe(""); expect(fromOpenAIResponse({}).done).toBe(""); diff --git a/tests/scoped-qa.test.ts b/tests/scoped-qa.test.ts index 400634e..4e58f26 100644 --- a/tests/scoped-qa.test.ts +++ b/tests/scoped-qa.test.ts @@ -396,3 +396,43 @@ describe("parseScopedArgs", () => { expect(parseScopedArgs([...base, "--pr", "7", "--repo", "itsthelore/x"])).toMatchObject({ pr: 7, repo: "itsthelore/x" }); }); }); + +describe("runScopedQa — per-capability error isolation", () => { + it("a capability whose drive throws becomes its own error entry; siblings survive", async () => { + const throwingDrive: ScopedQaDeps["drive"] = (options: DriveOptions) => { + if (options.capabilityId === "REQ-B") { + return Promise.reject(new Error("model call failed twice: 502; retry: 502")); + } + const session: RecordedSession = { + ...(options.capabilityId !== undefined ? { capabilityId: options.capabilityId } : {}), + title: options.title, + startUrl: options.startUrl, + actions: [ + { type: "goto", url: options.startUrl }, + { type: "expectText", locator: { kind: "testId", testId: "status" }, text: "ok" }, + ], + }; + return Promise.resolve({ session, finished: true, stopReason: "finished", steps: 1 } satisfies DriveResult); + }; + const config: ProofkeeperConfig = { + capabilities: [ + { id: "REQ-B", paths: ["src/b/**"], url: "http://b/" }, + { id: "REQ-C", paths: ["src/c/**"], url: "http://c/" }, + ], + }; + const deps: ScopedQaDeps = { drive: throwingDrive, makeCompiler: () => new FakeCompiler(), makeRunner: () => new FakeRunner("passed") }; + const result = await runScopedQa(deps, { + graph: GRAPH, + config, + changedPaths: ["src/b/y.ts", "src/c/z.ts"], + targetName: "local", + n: 1, + }); + + const byId = Object.fromEntries(result.driven.map((d) => [d.capability.id, d])); + expect(byId["REQ-B"]?.error).toMatch(/model call failed twice/); + expect(byId["REQ-B"]?.result).toBeUndefined(); + // The sibling completed and its result was not discarded. + expect(byId["REQ-C"]?.result?.verified).toBe(true); + }); +}); diff --git a/tests/terminal-recorder.test.ts b/tests/terminal-recorder.test.ts index 75df016..99e71e8 100644 --- a/tests/terminal-recorder.test.ts +++ b/tests/terminal-recorder.test.ts @@ -2,6 +2,7 @@ import { describe, expect, it } from "vitest"; import type { Page } from "@playwright/test"; import { Recorder } from "../src/compiler/recorder.js"; +import { runCommand } from "../src/compiler/terminal.js"; /** * The terminal half of the recorder runs real commands via spawnSync and never @@ -74,3 +75,9 @@ describe("Recorder — terminal actions", () => { await expect(rec.expectExit(0)).rejects.toThrow(/before any run_command/); }); }); + +describe("runCommand timeout", () => { + it("errors instead of hanging when a command outlives the cap", () => { + expect(() => runCommand("sleep 5", { timeoutMs: 150 })).toThrow(/ETIMEDOUT/); + }); +});