diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9d60853..04c4a5d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -54,6 +54,7 @@ jobs: tests/terminal.integration.test.ts tests/http.integration.test.ts tests/qa.integration.test.ts + tests/triage.integration.test.ts tests/plan.integration.test.ts tests/observe.integration.test.ts tests/extension.integration.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f8024e4..4b90861 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,15 @@ match exactly, or the workflow fails the publish. ## Unreleased +- **Triage a failing test: stale test or broken product.** New `proofkeeper + triage` command — when a `## Verified By` test goes red, it confirms the + failure, re-drives the capability against the requirement, and adjudicates: + **stale test** (exit 0 — the capability still verifies; a stable repair + candidate is compiled), **product regression** (exit 1 — the requirement's + outcomes could not be driven, with the reason as evidence), or + **inconclusive** (exit 3 — infrastructure noise, never a guessed verdict). + `--json` emits the machine contract; regressions feed failure-learning. + - **The drive has a trust boundary.** Everything the model observes is content from the product under test — untrusted input. The loop now enforces a policy on the model's tool calls: the **shell is off by default** (`run_command` diff --git a/README.md b/README.md index 01a15d2..cdc12f8 100644 --- a/README.md +++ b/README.md @@ -107,6 +107,22 @@ OPENAI_API_KEY=… GITHUB_TOKEN=… proofkeeper qa \ `proofkeeper qa` (alias `verify`) runs the whole loop for one capability: pick → drive → compile → fidelity → run → optionally propose the write-back. With `--config` it scopes to a pull request, driving every capability the changed files touch — concurrently, context-isolated — and posting one comment that updates in place. `--plan` has the model write a test plan first. +## Triage a failing test + +When a committed `## Verified By` test goes red, the question that matters is *which thing broke*. Because the requirement is ground truth, Proofkeeper can adjudicate: + +```bash +proofkeeper triage --corpus path/to/rac/ --capability REQ-CHECKOUT --url http://localhost:3000/ +``` + +It re-runs the failing spec to confirm, re-drives the capability against the requirement, and issues a verdict — the exit code is the contract: + +- **`0` — stale test.** The capability still verifies; the UI moved under the spec. A stable repair candidate is compiled and its path printed (an automated repair PR is planned). +- **`1` — product regression.** The requirement's outcomes could no longer be driven; the reason (the model's honest give-up) is the evidence. +- **`3` — inconclusive.** Infrastructure got in the way (model outage, runner errors, an unstable repair) — never a guessed verdict. + +`--json` emits the machine-readable verdict for CI. Agents without a requirements corpus can update a broken test or escalate it — they can't tell you whether the product still does what it's supposed to. + ## Bring your own model No model is bundled. Two adapters cover most providers; the `ModelClient` interface covers the rest. diff --git a/lore-proofkeeper/designs/design-regression-triage.md b/lore-proofkeeper/designs/design-regression-triage.md new file mode 100644 index 0000000..ebecfa9 --- /dev/null +++ b/lore-proofkeeper/designs/design-regression-triage.md @@ -0,0 +1,104 @@ +--- +schema_version: 1 +id: PK-KWGPQ30SFT0X +type: design +--- +# Triage Verdict Core — Confirm, Re-Drive, Adjudicate + +## Context + +Slice 1 of the regression-triage roadmap: the adjudication core behind +`proofkeeper triage`. The whole pipeline already exists — graph loading, +capability selection with `verifiedBy` paths, the drive seam, the compiler, +the fidelity gate, failure learning — so this design is one orchestrator and +one command over reused parts, plus the verdict model that makes the outcome +trustworthy. + +## User Need + +A maintainer looking at a red verifying test needs to know, without hand +investigation, whether to fix the test or fix the product — with evidence +either way, and an honest "cannot tell" when infrastructure got in the way. + +## Design + +- **Verdict model** (`src/triage/triage.ts`): a discriminated union — + `not_reproducible`, `stale_test` (carries the compiled repair candidate and + its fidelity verdict), `product_regression` (carries the reason), and + `inconclusive` (carries the reason). `runTriage` orchestrates: + 1. resolve the verifying spec (`resolveVerifier`: one entry ⇒ implicit, + several ⇒ `--spec` required, wrong spec ⇒ named error); + 2. confirm — re-run the old spec; passing ⇒ `not_reproducible`; + 3. re-drive with the requirement-derived goal, threading prior failures and + the trust boundary exactly as `runQa` does; + 4. adjudicate — finished + asserted + stable candidate ⇒ `stale_test`; + gave up / budget / assertion-free ⇒ `product_regression`; everything + else (runner error on the old spec, drive threw, candidate errored or + failed to stabilize) ⇒ `inconclusive`. +- **Conservatism is the invariant.** The two actionable verdicts are only + issued on clear evidence; instability and infrastructure noise never + masquerade as either. A fresh spec that held live but flakes under the gate + is `inconclusive`, not a regression. +- **CLI** (`src/cli.ts`): `parseTriageArgs` (pure, exported, `requireValue` + guards like every sibling), human rendering (verdict line + evidence + + next-step hint), `--json` stable contract, and `triageExitCode` — 0 stale / + not-reproducible, 1 regression, 2 usage (via `TriageInputError` → + `UsageError`), 3 inconclusive. Documented in USAGE. +- **Learning**: regression and inconclusive reasons are recorded with a + `[triage:]` prefix so future drives and reports can distinguish + triage-sourced knowledge. + +## Constraints + +- Reuse only: no new drive, compiler, gate, or runner behavior; the + orchestrator composes existing seams and stays unit-testable with the same + doubles as `runQa`. +- `runTriage` never throws for drive/runner failures — those are verdicts; + it throws only `TriageInputError` (caller-fixable) and programming errors. +- Exit code 3 extends the CLI contract additively; 0/1/2 keep their meanings. + +## Rationale + +Confirming the failure before re-driving keeps the expensive, model-driven +step off the path for specs that pass locally (mis-configured CI, stale +checkout). Adjudicating on "finished + asserted + stable" reuses the exact +bar `runQa` sets for "verified", so a stale-test verdict means precisely +"this capability would verify afresh today". + +## Alternatives + +- **Diff the old and new specs to classify the failure.** Rejected for the + verdict: spec text similarity is not evidence about the product; the live + re-drive is. (A diff is useful *presentation* for the slice-2 repair PR.) +- **Auto-replace the stale spec immediately.** Rejected: write-backs are + human-reviewed PRs (the trust boundary); slice 2 adds that path. +- **Treat an unstable repair as regression.** Rejected: the drive's + assertions held live at record time, so instability is flake evidence, not + product evidence. + +## Accessibility + +Not applicable — CLI text; the verdict line leads and the evidence follows. + +## Style Guidance + +Verdict words are loud and unambiguous (STALE TEST / PRODUCT REGRESSION / +INCONCLUSIVE); reasons are complete sentences carrying the model's own words +where they are the evidence. + +## Open Questions + +- Whether triage should accept a CI report (file of failing specs) and batch + over capabilities. Deferred to a later slice alongside the repair PR. + +## Related Requirements + +- req-regression-triage + +## Related Roadmaps + +- regression-triage + +## Status + +Accepted diff --git a/lore-proofkeeper/requirements/req-regression-triage.md b/lore-proofkeeper/requirements/req-regression-triage.md new file mode 100644 index 0000000..1deb79b --- /dev/null +++ b/lore-proofkeeper/requirements/req-regression-triage.md @@ -0,0 +1,62 @@ +--- +schema_version: 1 +id: PK-KWGPQ2742QKS +type: requirement +--- +# Triage Verdict — Stale Test or Broken Product + +## Problem + +A red verifying test only says something broke — not *which thing*. Without +adjudication, a maintainer investigates every failure by hand, and the two +failure classes demand opposite responses: a stale test needs a repaired spec, +a product regression needs a bug fix. QA agents without a requirements corpus +cannot make this distinction; Proofkeeper can, because the requirement text is +ground truth a fresh drive can be judged against. This requirement covers +slice 1 of the regression-triage roadmap: the verdict core and the CLI +command. The automated repair PR and the regression issue report follow in a +later slice. + +## Requirements + +- [REQ-001] `proofkeeper triage` takes a coverage source, a capability id, and a target URL; the capability's verifying spec is resolved from its `verified_by` edges, with `--spec` required (and validated) only when there are several. +- [REQ-002] The failure is confirmed first: the verifying spec is re-run against the target, and a passing spec yields a `not_reproducible` verdict with no re-drive. +- [REQ-003] A failing spec triggers a re-drive of the capability with the requirement-derived goal; a drive that finishes, asserts, and compiles to a spec that passes the fidelity gate yields `stale_test` with the repair candidate's path and fidelity as evidence. +- [REQ-004] A re-drive that gives up, exhausts its step budget, or asserts nothing yields `product_regression` with the honest reason as evidence. +- [REQ-005] Anything muddied by infrastructure — the verifying spec cannot run, the re-drive itself fails, the repair candidate never runs cleanly or does not stabilize — yields `inconclusive`, never a guessed verdict. +- [REQ-006] The exit code is the machine contract: 0 for stale test / not reproducible, 1 for product regression, 2 for usage errors, 3 for inconclusive; `--json` emits the stable structured verdict. +- [REQ-007] `product_regression` and `inconclusive` reasons are recorded to the failure-learning store; verifying outcomes record nothing. + +## Success Metrics + +- A renamed control (requirement unchanged) adjudicates as `stale_test` with a + stable repair candidate, end to end in a real browser. +- A broken flow (outcome never reachable) adjudicates as `product_regression` + naming the model's give-up reason. +- Exit codes 0/1/3 are distinguishable in CI without parsing output. + +## Risks + +- A model could mis-drive a healthy product into a false regression verdict. + Mitigation: prior failure learning steers the re-drive, the verdict carries + the full reason for human review, and the fidelity gate guards the opposite + direction (no false stale-test without a stable repair). +- Confirming the failure re-runs a known-failing spec, which can be slow + (action timeouts). Mitigation: bounded by the runner's existing wall-clock + caps. + +## Assumptions + +- The requirement-derived goal (`defaultGoal`) carries enough observable + intent to adjudicate against; ambiguity degrades to `inconclusive`. +- The trust boundary and resilience behavior of the ordinary drive apply + unchanged to re-drives. + +## Related Roadmaps + +- regression-triage + +## Verified By + +- `tests/triage.test.ts` +- `tests/triage.integration.test.ts` diff --git a/src/cli.ts b/src/cli.ts index fcad1a6..549d09a 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -30,6 +30,7 @@ import { GitHubRestGateway } from "./writeback/gateways/github-rest.js"; import { GitHubWriteBackProposer, type WriteBackProposer } from "./writeback/proposer.js"; import { renderScopedQaComment, upsertComment, SCOPED_QA_MARKER, type ScopedQaCommentInput, type ScopedQaCommentRow } from "./writeback/comment.js"; import { scaffoldConfig, renderScaffoldedConfig } from "./scaffold/scaffold.js"; +import { runTriage, TriageInputError, type TriageDeps, type TriageResult } from "./triage/triage.js"; import { execFile } from "node:child_process"; import { readFile, writeFile, stat } from "node:fs/promises"; @@ -41,6 +42,8 @@ const execFileAsync = promisify(execFile); const EXIT_OK = 0; const EXIT_UNVERIFIED = 1; const EXIT_USAGE = 2; +/** Triage could not reach a confident verdict (infra failure, unstable repair). */ +const EXIT_INCONCLUSIVE = 3; const USAGE = `proofkeeper — autonomous verification for the Lore family @@ -50,6 +53,8 @@ Usage: proofkeeper qa (--graph-file | --corpus ) --url [options] proofkeeper qa (--graph-file | --corpus ) --config (--changed | --base-ref ) [options] + proofkeeper triage (--graph-file | --corpus ) --capability + --url [options] proofkeeper --help Commands: @@ -61,6 +66,11 @@ Commands: (optionally) propose the Verified By write-back. Alias: verify. With --config, scope to a change: drive every unverified capability the changed files touch and post the evidence to a pull request. + triage Adjudicate a failing Verified By test against its requirement: + re-run the spec to confirm, re-drive the capability, and report + STALE TEST (capability still verifies — repair candidate compiled), + PRODUCT REGRESSION (its outcomes can no longer be driven), or + INCONCLUSIVE. The exit code is the verdict. Coverage options: --graph-file Read a 'rac export --graph' JSON file (primary). @@ -97,6 +107,18 @@ qa options: --repo Target repository for the write-back (required with --propose). --base Base branch the write-back PR targets (default: main). +triage options: + --graph-file | --corpus Coverage source (one required). + --capability The capability whose verifying test failed (required). + --url Product entry point the re-drive starts from (required). + --spec Which verifying spec to adjudicate (required only when + the capability has more than one). + --goal Goal override (default: derived from the requirement). + --n Fidelity re-runs the repair candidate must pass (default: 3). + --json Emit the stable machine-readable verdict. + (--target-name, --base-url, --max-steps, --out-dir, --allow-shell, + --allow-host, --verbose as under qa.) + scoped qa options (with --config): --config Path map: which capabilities each changed file touches. --changed Comma-separated changed files (else --base-ref). @@ -119,9 +141,12 @@ Options: --version, -v Print the version. Exit codes: - 0 success (everything verified, or the driven test is stable) - 1 not verified (unverified capabilities, or an unstable test) + 0 success (everything verified, the driven test is stable, or triage found + a stale test / could not reproduce the failure) + 1 not verified (unverified capabilities, an unstable test, or a triage + PRODUCT REGRESSION verdict) 2 usage or parse error + 3 triage was inconclusive (infrastructure failure or an unstable repair) `; class UsageError extends Error {} @@ -806,6 +831,233 @@ async function runScopedCommand(argv: string[]): Promise { return anyUnverified ? EXIT_UNVERIFIED : EXIT_OK; } +// --------------------------------------------------------------------------- +// triage +// --------------------------------------------------------------------------- + +export interface TriageArgs { + graphFile?: string; + corpus?: string; + capability: string; + spec?: string; + url: string; + goal?: string; + targetName: string; + baseUrl: string; + n: number; + maxSteps?: number; + outDir: string; + allowShell: boolean; + allowedHosts: string[]; + verbose: boolean; + json: boolean; +} + +/** Parse `triage` arguments. Pure and exported so it is unit-testable. */ +export function parseTriageArgs(argv: string[]): TriageArgs { + const raw: Partial = {}; + for (let i = 0; i < argv.length; i++) { + const arg = argv[i]; + switch (arg) { + case "--graph-file": + raw.graphFile = requireValue(argv[++i], "--graph-file"); + break; + case "--corpus": + raw.corpus = requireValue(argv[++i], "--corpus"); + break; + case "--capability": + raw.capability = requireValue(argv[++i], "--capability"); + break; + case "--spec": + raw.spec = requireValue(argv[++i], "--spec"); + break; + case "--url": + raw.url = requireValue(argv[++i], "--url"); + break; + case "--goal": + raw.goal = requireValue(argv[++i], "--goal"); + break; + case "--target-name": + raw.targetName = requireValue(argv[++i], "--target-name"); + break; + case "--base-url": + raw.baseUrl = requireValue(argv[++i], "--base-url"); + break; + case "--n": + raw.n = parsePositiveInt(argv[++i], "--n"); + break; + case "--max-steps": + raw.maxSteps = parsePositiveInt(argv[++i], "--max-steps"); + break; + case "--out-dir": + raw.outDir = requireValue(argv[++i], "--out-dir"); + break; + case "--allow-shell": + raw.allowShell = true; + break; + case "--allow-host": + raw.allowedHosts ??= []; + raw.allowedHosts.push(requireValue(argv[++i], "--allow-host")); + break; + case "--verbose": + raw.verbose = true; + break; + case "--json": + raw.json = true; + break; + default: + throw new UsageError(`unknown option '${arg}'`); + } + } + + if (!raw.graphFile && !raw.corpus) { + throw new UsageError("triage requires --graph-file or --corpus "); + } + if (raw.graphFile && raw.corpus) { + throw new UsageError("pass only one of --graph-file or --corpus"); + } + if (!raw.capability) throw new UsageError("triage requires --capability "); + if (!raw.url) throw new UsageError("triage requires --url "); + + return { + ...(raw.graphFile !== undefined ? { graphFile: raw.graphFile } : {}), + ...(raw.corpus !== undefined ? { corpus: raw.corpus } : {}), + capability: raw.capability, + ...(raw.spec !== undefined ? { spec: raw.spec } : {}), + url: raw.url, + ...(raw.goal !== undefined ? { goal: raw.goal } : {}), + targetName: raw.targetName ?? "local", + baseUrl: raw.baseUrl ?? raw.url, + n: raw.n ?? 3, + ...(raw.maxSteps !== undefined ? { maxSteps: raw.maxSteps } : {}), + outDir: raw.outDir ?? "tests/generated", + allowShell: raw.allowShell ?? false, + allowedHosts: raw.allowedHosts ?? [], + verbose: raw.verbose ?? false, + json: raw.json ?? false, + }; +} + +/** Exit code for a triage verdict: the code IS the machine contract. */ +export function triageExitCode(result: TriageResult): number { + switch (result.verdict.kind) { + case "not_reproducible": + case "stale_test": + return EXIT_OK; + case "product_regression": + return EXIT_UNVERIFIED; + case "inconclusive": + return EXIT_INCONCLUSIVE; + } +} + +/** Render a triage result for humans: the verdict, then the evidence. */ +export function renderTriageResult(result: TriageResult): string { + const lines = [ + `Capability: ${result.capability.id} — ${result.capability.title}`, + `Verifying spec: ${result.oldSpec}` + + (result.oldRun ? ` (re-run: ${result.oldRun.status})` : " (could not run)"), + ]; + if (result.drive) { + const stop = + result.drive.stopReason === "finished" + ? "finished" + : result.drive.stopReason === "gave_up" + ? "gave up" + : "stopped at step budget"; + lines.push(`Re-drive: ${result.drive.steps} step(s), ${stop}`); + } + switch (result.verdict.kind) { + case "not_reproducible": + lines.push("Verdict: NOT REPRODUCIBLE — the verifying spec passes against this target; nothing to triage."); + break; + case "stale_test": + lines.push( + "Verdict: STALE TEST — the capability still verifies; the committed spec is out of date.", + `Repair candidate: ${result.verdict.candidate.specPath}`, + `Fidelity: ${result.verdict.fidelity.passed}/${result.verdict.fidelity.attempts} re-runs green`, + "Next: review the candidate and replace the stale spec (automated repair PR lands in a later slice).", + ); + break; + case "product_regression": + lines.push( + "Verdict: PRODUCT REGRESSION — the requirement's outcomes could not be driven.", + `Reason: ${result.verdict.reason}`, + ); + break; + case "inconclusive": + lines.push("Verdict: INCONCLUSIVE — no confident adjudication.", `Reason: ${result.verdict.reason}`); + break; + } + return lines.join("\n"); +} + +/** The stable machine contract for `triage --json`. */ +export function triageToJson(result: TriageResult): string { + return JSON.stringify( + { + capability: { id: result.capability.id, title: result.capability.title }, + oldSpec: result.oldSpec, + ...(result.oldRun !== undefined ? { oldRunStatus: result.oldRun.status } : {}), + verdict: result.verdict.kind, + ...(result.verdict.kind === "product_regression" || result.verdict.kind === "inconclusive" + ? { reason: result.verdict.reason } + : {}), + ...(result.verdict.kind === "stale_test" + ? { + candidateSpec: result.verdict.candidate.specPath, + fidelity: { passed: result.verdict.fidelity.passed, attempts: result.verdict.fidelity.attempts }, + } + : {}), + ...(result.drive !== undefined + ? { drive: { steps: result.drive.steps, stopReason: result.drive.stopReason } } + : {}), + }, + null, + 2, + ); +} + +async function runTriageCommand(argv: string[]): Promise { + const args = parseTriageArgs(argv); + const model = resolveModel(); + + const graph = args.graphFile + ? await loadGraphFromFile(args.graphFile) + : await loadGraphFromCorpus(args.corpus!); + + const target: RunTarget = { name: args.targetName, baseURL: args.baseUrl }; + const deps: TriageDeps = { + drive: browserDrive(model, { verbose: args.verbose }), + compiler: new CodegenCompiler({ outDir: args.outDir }), + runner: new PlaywrightRunner(), + learning: new FileLearningStore(), + }; + + let result: TriageResult; + try { + result = await runTriage(deps, { + graph, + capabilityId: args.capability, + ...(args.spec !== undefined ? { spec: args.spec } : {}), + startUrl: args.url, + ...(args.goal !== undefined ? { goal: args.goal } : {}), + target, + n: args.n, + ...(args.maxSteps !== undefined ? { maxSteps: args.maxSteps } : {}), + ...(args.allowShell ? { allowShell: true } : {}), + ...(args.allowedHosts.length > 0 ? { allowedHosts: args.allowedHosts } : {}), + }); + } catch (err) { + // Caller-fixable input problems share the usage exit code. + if (err instanceof TriageInputError) throw new UsageError(err.message); + throw err; + } + + process.stdout.write((args.json ? triageToJson(result) : renderTriageResult(result)) + "\n"); + return triageExitCode(result); +} + // --------------------------------------------------------------------------- // dispatch // --------------------------------------------------------------------------- @@ -840,6 +1092,8 @@ export async function main(argv: string[]): Promise { case "qa": case "verify": return await runQaCommand(rest); + case "triage": + return await runTriageCommand(rest); default: process.stderr.write(`unknown command '${command}'\n\n${USAGE}`); return EXIT_USAGE; diff --git a/src/index.ts b/src/index.ts index f71bbf5..59b641a 100644 --- a/src/index.ts +++ b/src/index.ts @@ -103,6 +103,10 @@ export type { PageObservation, PageMonitor } from "./agent/observe.js"; export { loadExtension, extensionIdFromUrl } from "./agent/extension.js"; export type { LoadedExtension } from "./agent/extension.js"; +// Regression triage — stale test or broken product (roadmap: regression-triage). +export { runTriage, resolveVerifier, TriageInputError } from "./triage/triage.js"; +export type { TriageDeps, TriageOptions, TriageResult, TriageVerdict } from "./triage/triage.js"; + // The drive's trust boundary — egress policy and observation redaction. export { buildPolicy, urlRefusal, callRefusal, SHELL_TOOL_NAMES } from "./agent/policy.js"; export type { EgressPolicy } from "./agent/policy.js"; diff --git a/src/triage/triage.ts b/src/triage/triage.ts new file mode 100644 index 0000000..7aae8cd --- /dev/null +++ b/src/triage/triage.ts @@ -0,0 +1,246 @@ +/** + * Regression triage — adjudicate a failing verifying test against its + * requirement (roadmap: regression-triage, slice 1). + * + * When a committed `## Verified By` test fails, the interesting question is + * not "is it red" but "*which thing broke*": the test (the UI changed, the + * requirement still holds) or the product (the requirement's outcomes can no + * longer be driven). Proofkeeper can answer because the requirement is ground + * truth: re-drive the capability with the requirement-derived goal and let + * the outcome adjudicate. + * + * The verdict is deliberately conservative: `stale_test` and + * `product_regression` are only issued on clear evidence; anything muddied by + * infrastructure (model outage, runner errors, an unstable fresh spec) is + * `inconclusive` — never a guess. Slice 2 turns a stale-test verdict into a + * repair PR through the existing human-reviewed write-back path. + */ + +import type { Graph } from "../coverage/graph.js"; +import type { CapabilityCoverage } from "../coverage/model.js"; +import { selectCapability, defaultGoal } from "../qa/run-qa.js"; +import { sessionAssertsOutcome } from "../compiler/actions.js"; +import type { CandidateTest, Compiler } from "../compiler/types.js"; +import { assessFidelity, type FidelityVerdict } from "../fidelity/gate.js"; +import type { Runner, RunResult, RunTarget } from "../runner/types.js"; +import type { DriveOptions, DriveResult } from "../agent/drive.js"; +import type { LearningStore } from "../learning/store.js"; + +/** Raised for caller-fixable input problems (maps to CLI exit 2). */ +export class TriageInputError extends Error { + constructor(message: string) { + super(message); + this.name = "TriageInputError"; + } +} + +/** The adjudication outcome for one failing verifying test. */ +export type TriageVerdict = + /** The old spec passes against this target now — nothing to triage. */ + | { kind: "not_reproducible" } + /** + * The capability still verifies: the re-drive finished, asserted the + * requirement's outcomes, and its compiled spec is stable. The old spec is + * out of date; `candidate` is the repair. + */ + | { kind: "stale_test"; candidate: CandidateTest; fidelity: FidelityVerdict } + /** The requirement's outcomes could not be driven — the product broke. */ + | { kind: "product_regression"; reason: string } + /** Infrastructure or instability prevented a confident verdict. */ + | { kind: "inconclusive"; reason: string }; + +export interface TriageDeps { + /** Drive the product to re-record the capability (same seam as {@link QaDeps}). */ + drive(options: DriveOptions): Promise; + compiler: Compiler; + runner: Runner; + /** Optional: regression/inconclusive reasons feed the failure-learning store. */ + learning?: LearningStore; +} + +export interface TriageOptions { + /** Parsed `rac export --graph` output. */ + graph: Graph; + /** The capability whose verifying test failed. */ + capabilityId: string; + /** + * The failing spec's path. Optional when the capability has exactly one + * `verified_by` entry; required (and validated) when it has several. + */ + spec?: string; + /** Product entry point the re-drive navigates to first. */ + startUrl: string; + /** Goal override; defaults to the requirement-derived {@link defaultGoal}. */ + goal?: string; + /** Target the old spec and the fresh candidate run against. */ + target: RunTarget; + /** Fidelity re-run count for the repair candidate. */ + n: number; + maxSteps?: number; + allowShell?: boolean; + allowedHosts?: string[]; +} + +export interface TriageResult { + capability: CapabilityCoverage; + /** The verifying spec that was adjudicated. */ + oldSpec: string; + /** The confirming run of the old spec (absent when the runner errored). */ + oldRun?: RunResult; + verdict: TriageVerdict; + /** The re-drive, when one ran. */ + drive?: DriveResult; +} + +/** + * Resolve which verifying spec to adjudicate: an explicit `spec` must be one + * of the capability's verifiers; otherwise the capability must have exactly + * one. + * + * @throws {TriageInputError} on no verifiers, an ambiguous choice, or a spec + * that does not verify this capability. + */ +export function resolveVerifier(capability: CapabilityCoverage, spec?: string): string { + if (capability.verifiedBy.length === 0) { + throw new TriageInputError( + `capability '${capability.id}' has no verifying test to triage — run 'proofkeeper qa' to create one`, + ); + } + if (spec !== undefined) { + if (!capability.verifiedBy.includes(spec)) { + throw new TriageInputError( + `'${spec}' does not verify '${capability.id}' — its verifiers are: ${capability.verifiedBy.join(", ")}`, + ); + } + return spec; + } + if (capability.verifiedBy.length > 1) { + throw new TriageInputError( + `capability '${capability.id}' has ${capability.verifiedBy.length} verifying tests — ` + + `pass --spec to choose one of: ${capability.verifiedBy.join(", ")}`, + ); + } + return capability.verifiedBy[0]!; +} + +/** + * Adjudicate one failing verifying test: confirm the failure, re-drive the + * capability against its requirement, and issue a {@link TriageVerdict}. + * Never throws on a drive/runner failure — that is an `inconclusive` verdict. + */ +export async function runTriage(deps: TriageDeps, options: TriageOptions): Promise { + const capability = selectCapability(options.graph, options.capabilityId); + const oldSpec = resolveVerifier(capability, options.spec); + + // 1. CONFIRM — re-run the old spec; a passing spec has nothing to triage. + let oldRun: RunResult | undefined; + try { + const results = await deps.runner.run([{ id: `triage-old-${capability.id}`, specPath: oldSpec }], { + targets: [options.target], + }); + oldRun = results[0]; + } catch (err) { + return { + capability, + oldSpec, + verdict: { + kind: "inconclusive", + reason: `could not run the verifying spec: ${(err as Error).message}`, + }, + }; + } + if (oldRun !== undefined && oldRun.status === "passed") { + return { capability, oldSpec, oldRun, verdict: { kind: "not_reproducible" } }; + } + + // 2. RE-DRIVE — the requirement text is the goal; the drive is the judge. + const goal = options.goal ?? defaultGoal(capability); + const prior = deps.learning ? await deps.learning.priorFailures(capability.id) : []; + const driveOptions: DriveOptions = { + capabilityId: capability.id, + title: `verify ${capability.title}`, + startUrl: options.startUrl, + goal, + ...(options.maxSteps !== undefined ? { maxSteps: options.maxSteps } : {}), + ...(prior.length > 0 ? { priorFailures: prior.map((f) => f.reason) } : {}), + ...(options.allowShell !== undefined ? { allowShell: options.allowShell } : {}), + ...(options.allowedHosts !== undefined ? { allowedHosts: options.allowedHosts } : {}), + }; + + let drive: DriveResult; + try { + drive = await deps.drive(driveOptions); + } catch (err) { + // A model/browser outage says nothing about the product. + return { + capability, + oldSpec, + ...(oldRun !== undefined ? { oldRun } : {}), + verdict: { kind: "inconclusive", reason: `re-drive failed to run: ${(err as Error).message}` }, + }; + } + + // 3. ADJUDICATE. + const base = { capability, oldSpec, ...(oldRun !== undefined ? { oldRun } : {}), drive }; + + if (!drive.finished || !sessionAssertsOutcome(drive.session)) { + // The model could not drive and assert the requirement's outcomes on the + // live product: the requirement is violated as far as a drive can tell. + const reason = !drive.finished + ? drive.stopReason === "gave_up" + ? `re-drive gave up after ${drive.steps} step(s)` + + (drive.gaveUpText !== undefined ? `: ${drive.gaveUpText}` : "") + : `re-drive did not finish within the step budget (${drive.steps} steps)` + : "re-drive finished but could not assert any observable outcome"; + return await recorded(deps, capability.id, goal, drive.steps, { + ...base, + verdict: { kind: "product_regression", reason }, + }); + } + + let candidate: CandidateTest; + try { + candidate = await deps.compiler.compile(drive.session); + } catch (err) { + return await recorded(deps, capability.id, goal, drive.steps, { + ...base, + verdict: { kind: "inconclusive", reason: `repair candidate failed to compile: ${(err as Error).message}` }, + }); + } + + const fidelity = await assessFidelity(deps.runner, candidate, { n: options.n, target: options.target }); + if (fidelity.stable) { + // The capability verifies afresh: the product is fine, the old spec is stale. + return { ...base, verdict: { kind: "stale_test", candidate, fidelity } }; + } + + // The re-drive held live but its spec did not stabilize — no confident + // verdict either way (flake or infra, not proven regression). + const reason = + fidelity.errors !== undefined && fidelity.passed === 0 + ? `repair candidate never ran cleanly: ${fidelity.errors.join("; ")}` + : `re-drive succeeded but its spec did not stabilize (${fidelity.passed}/${fidelity.attempts} green)`; + return await recorded(deps, capability.id, goal, drive.steps, { + ...base, + verdict: { kind: "inconclusive", reason }, + }); +} + +/** Record a non-verifying triage outcome to failure-learning, then return it. */ +async function recorded( + deps: TriageDeps, + capabilityId: string, + goal: string, + steps: number, + result: TriageResult, +): Promise { + if (deps.learning && (result.verdict.kind === "product_regression" || result.verdict.kind === "inconclusive")) { + await deps.learning.recordFailure({ + capabilityId, + goal, + reason: `[triage:${result.verdict.kind}] ${result.verdict.reason}`, + steps, + }); + } + return result; +} diff --git a/tests/triage.integration.test.ts b/tests/triage.integration.test.ts new file mode 100644 index 0000000..e5f095f --- /dev/null +++ b/tests/triage.integration.test.ts @@ -0,0 +1,186 @@ +import { createServer, type Server } from "node:http"; +import { fileURLToPath } from "node:url"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { chromium, type Browser } from "@playwright/test"; + +import { AutonomousDriver } from "../src/agent/drive.js"; +import type { ModelClient, ModelRequest, ModelResponse } from "../src/agent/model.js"; +import { CodegenCompiler } from "../src/compiler/compiler.js"; +import { PlaywrightRunner } from "../src/runner/playwright-runner.js"; +import { runTriage, type TriageDeps } from "../src/triage/triage.js"; +import type { Graph } from "../src/coverage/graph.js"; + +/** + * Regression triage end to end (roadmap: regression-triage): a committed spec + * recorded against yesterday's UI fails against today's — and triage + * adjudicates it in a real browser. + * + * - The UI renamed its button (requirement still met) → STALE TEST, with a + * stable repair candidate compiled from the fresh drive. + * - The product broke (the status never flips) → PRODUCT REGRESSION. + * + * Gated behind PROOFKEEPER_E2E like the other real-browser suites. + */ +const e2e = process.env.PROOFKEEPER_E2E ? describe : describe.skip; + +const projectRoot = fileURLToPath(new URL("..", import.meta.url)); + +/** The product page, parameterized: yesterday's button name, and whether clicking works. */ +function productHtml(buttonLabel: string, working: boolean): string { + return ` +

Triage Demo

+ +

unverified

+ + `; +} + +/** Decides from observations: click whichever button the page offers, then assert + finish. */ +class AdaptiveVerifyModel implements ModelClient { + complete(request: ModelRequest): Promise { + const last = [...request.transcript].reverse().find((m) => m.role === "user")?.content ?? ""; + if (last.includes("verified") && !last.includes("unverified")) { + return Promise.resolve({ + toolCalls: [ + { name: "expect_text", arguments: { locator: { strategy: "testId", testId: "status" }, text: "verified" } }, + { name: "finish", arguments: {} }, + ], + }); + } + const label = last.includes("Confirm") ? "Confirm" : "Verify"; + return Promise.resolve({ + toolCalls: [{ name: "click", arguments: { locator: { strategy: "role", role: "button", name: label } } }], + }); + } +} + +/** Clicks once; when the status never flips, honestly gives up. */ +class GivesUpModel implements ModelClient { + private clicked = false; + complete(_request: ModelRequest): Promise { + if (!this.clicked) { + this.clicked = true; + return Promise.resolve({ + toolCalls: [{ name: "click", arguments: { locator: { strategy: "role", role: "button", name: "Verify" } } }], + }); + } + return Promise.resolve({ done: "The status never changes to verified; the flow appears broken." }); + } +} + +e2e("triage — a failing verifying spec is adjudicated in a real browser", () => { + let server: Server; + let baseURL: string; + let browser: Browser; + let currentHtml = productHtml("Verify", true); + let oldSpecPath: string; + let graph: Graph; + + const driveWith = + (model: ModelClient): TriageDeps["drive"] => + async (options) => { + const page = await browser.newPage(); + try { + return await new AutonomousDriver(page, model, options).drive(); + } finally { + await page.close(); + } + }; + + beforeAll(async () => { + server = createServer((_req, res) => { + res.writeHead(200, { "content-type": "text/html" }); + res.end(currentHtml); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const addr = server.address(); + if (typeof addr === "string" || addr === null) throw new Error("no server address"); + baseURL = `http://127.0.0.1:${addr.port}/`; + browser = await chromium.launch(); + + // Record "yesterday's" committed spec against the original UI. + const drive = await driveWith(new AdaptiveVerifyModel())({ + capabilityId: "REQ-TRIAGE", + title: "verify flips status to verified", + startUrl: baseURL, + goal: "Click the button and confirm the status changes to 'verified'.", + }); + expect(drive.finished).toBe(true); + const candidate = await new CodegenCompiler({ outDir: "examples/generated/triage/old" }).compile(drive.session); + oldSpecPath = candidate.specPath; + + graph = { + schema_version: "1", + source: "triage-demo", + nodes: [{ id: "REQ-TRIAGE", type: "requirement", status: "Accepted", title: "Verify flow" }], + edges: [{ source: "REQ-TRIAGE", target: oldSpecPath, type: "verified_by", directed: true, resolved: false }], + }; + }, 300_000); + + afterAll(async () => { + await browser?.close(); + await new Promise((resolve) => server?.close(() => resolve())); + }); + + it( + "STALE TEST — the button was renamed, the requirement still holds", + async () => { + currentHtml = productHtml("Confirm", true); // today's UI: renamed, still working + + const deps: TriageDeps = { + drive: driveWith(new AdaptiveVerifyModel()), + compiler: new CodegenCompiler({ outDir: "examples/generated/triage/repair" }), + runner: new PlaywrightRunner({ cwd: projectRoot, outputDir: "test-results/triage" }), + }; + const result = await runTriage(deps, { + graph, + capabilityId: "REQ-TRIAGE", + startUrl: baseURL, + target: { name: "local", baseURL }, + n: 2, + }); + + // The committed spec really fails — by timeout here: the renamed button + // never appears, so the recorded click waits out its action timeout. + expect(result.oldRun?.status).not.toBe("passed"); + expect(result.verdict.kind).toBe("stale_test"); + if (result.verdict.kind === "stale_test") { + expect(result.verdict.fidelity.stable).toBe(true); + expect(result.verdict.candidate.specPath).toContain("triage/repair"); + } + }, + 300_000, + ); + + it( + "PRODUCT REGRESSION — the status never flips, the requirement is violated", + async () => { + currentHtml = productHtml("Verify", false); // today's UI: broken + + const deps: TriageDeps = { + drive: driveWith(new GivesUpModel()), + compiler: new CodegenCompiler({ outDir: "examples/generated/triage/broken" }), + runner: new PlaywrightRunner({ cwd: projectRoot, outputDir: "test-results/triage" }), + }; + const result = await runTriage(deps, { + graph, + capabilityId: "REQ-TRIAGE", + startUrl: baseURL, + target: { name: "local", baseURL }, + n: 2, + maxSteps: 4, + }); + + expect(result.oldRun?.status).not.toBe("passed"); + expect(result.verdict.kind).toBe("product_regression"); + if (result.verdict.kind === "product_regression") { + expect(result.verdict.reason).toContain("gave up"); + } + }, + 300_000, + ); +}); diff --git a/tests/triage.test.ts b/tests/triage.test.ts new file mode 100644 index 0000000..359a88f --- /dev/null +++ b/tests/triage.test.ts @@ -0,0 +1,259 @@ +/** + * Regression triage — the re-drive verdict core (roadmap: regression-triage). + * + * The verdict must be earned: stale_test and product_regression only on clear + * evidence; anything muddied by infrastructure is inconclusive. These tests + * pin every verdict path with fake runner/drive doubles, plus the CLI + * contract (parser, exit codes). + */ + +import { describe, expect, it } from "vitest"; + +import { runTriage, resolveVerifier, TriageInputError, type TriageDeps } from "../src/triage/triage.js"; +import { parseTriageArgs, triageExitCode, renderTriageResult, triageToJson } from "../src/cli.js"; +import { computeCoverage } from "../src/coverage/model.js"; +import { InMemoryLearningStore } from "../src/learning/store.js"; +import type { Graph } from "../src/coverage/graph.js"; +import type { RecordedSession } from "../src/compiler/actions.js"; +import type { CandidateTest, Compiler } from "../src/compiler/types.js"; +import type { CompiledTest, RunOptions, RunResult, RunStatus, Runner } from "../src/runner/types.js"; +import type { DriveOptions, DriveResult } from "../src/agent/drive.js"; + +const GRAPH: Graph = { + schema_version: "1", + source: "demo", + nodes: [ + { id: "REQ-A", type: "requirement", status: "Accepted", title: "Alpha" }, + { id: "REQ-TWO", type: "requirement", status: "Accepted", title: "Twice" }, + { id: "REQ-NONE", type: "requirement", status: "Accepted", title: "Bare" }, + ], + edges: [ + { source: "REQ-A", target: "tests/a.spec.ts", type: "verified_by", directed: true, resolved: false }, + { source: "REQ-TWO", target: "tests/t1.spec.ts", type: "verified_by", directed: true, resolved: false }, + { source: "REQ-TWO", target: "tests/t2.spec.ts", type: "verified_by", directed: true, resolved: false }, + ], +}; + +const TARGET = { name: "local", baseURL: "http://localhost:3000/" }; + +class FakeCompiler implements Compiler { + compile(session: RecordedSession): Promise { + return Promise.resolve({ + id: "cand", + specPath: "tests/generated/cand.spec.ts", + title: session.title, + fromSession: session, + }); + } +} + +/** + * A runner scripted per spec path: the old spec's status comes from `oldSpec`, + * every other run (the fidelity gate on the candidate) from `candidate` — + * which may be a status, a sequence, or "throw". + */ +class ScriptedRunner implements Runner { + private candidateCall = 0; + constructor( + private readonly oldSpec: RunStatus | "throw", + private readonly candidate: (RunStatus | "throw")[] = ["passed"], + ) {} + run(suite: CompiledTest[], _options: RunOptions): Promise { + const isOld = suite[0]!.id.startsWith("triage-old-"); + const script = isOld + ? this.oldSpec + : (this.candidate[Math.min(this.candidateCall++, this.candidate.length - 1)] ?? "passed"); + if (script === "throw") return Promise.reject(new Error("playwright run failed: browser hung")); + return Promise.resolve( + suite.map((t) => ({ testId: t.id, target: TARGET.name, status: script, durationMs: 1 })), + ); + } +} + +function drives(result: Partial & { actions?: RecordedSession["actions"] }): TriageDeps["drive"] { + return (options: DriveOptions) => + Promise.resolve({ + session: { + ...(options.capabilityId !== undefined ? { capabilityId: options.capabilityId } : {}), + title: options.title, + startUrl: options.startUrl, + actions: result.actions ?? [ + { type: "goto", url: options.startUrl }, + { type: "expectText", locator: { kind: "testId", testId: "status" }, text: "ok" }, + ], + }, + finished: result.finished ?? true, + stopReason: result.stopReason ?? "finished", + ...(result.gaveUpText !== undefined ? { gaveUpText: result.gaveUpText } : {}), + steps: result.steps ?? 3, + } satisfies DriveResult); +} + +const throwingDrive: TriageDeps["drive"] = () => + Promise.reject(new Error("model call failed twice: 502; retry: 502")); + +function deps(runner: Runner, drive: TriageDeps["drive"], learning?: InMemoryLearningStore): TriageDeps { + return { drive, compiler: new FakeCompiler(), runner, ...(learning ? { learning } : {}) }; +} + +const OPTIONS = { graph: GRAPH, capabilityId: "REQ-A", startUrl: "http://x/", target: TARGET, n: 2 }; + +describe("resolveVerifier", () => { + const coverage = computeCoverage(GRAPH); + const byId = (id: string) => [...coverage.verified, ...coverage.unverified].find((c) => c.id === id)!; + + it("uses the single verifier when the capability has exactly one", () => { + expect(resolveVerifier(byId("REQ-A"))).toBe("tests/a.spec.ts"); + }); + + it("requires --spec when there are several, naming them", () => { + expect(() => resolveVerifier(byId("REQ-TWO"))).toThrow(TriageInputError); + expect(() => resolveVerifier(byId("REQ-TWO"))).toThrow(/t1\.spec\.ts.*t2\.spec\.ts/); + expect(resolveVerifier(byId("REQ-TWO"), "tests/t2.spec.ts")).toBe("tests/t2.spec.ts"); + }); + + it("rejects a spec that does not verify the capability", () => { + expect(() => resolveVerifier(byId("REQ-A"), "tests/other.spec.ts")).toThrow(/does not verify/); + }); + + it("rejects a capability with no verifying test", () => { + expect(() => resolveVerifier(byId("REQ-NONE"))).toThrow(/no verifying test/); + }); +}); + +describe("runTriage verdicts", () => { + it("not_reproducible — the old spec passes against this target", async () => { + const result = await runTriage(deps(new ScriptedRunner("passed"), drives({})), OPTIONS); + expect(result.verdict.kind).toBe("not_reproducible"); + expect(result.oldRun?.status).toBe("passed"); + expect(result.drive).toBeUndefined(); + }); + + it("stale_test — the re-drive finishes, asserts, and its spec is stable", async () => { + const result = await runTriage(deps(new ScriptedRunner("failed", ["passed", "passed"]), drives({})), OPTIONS); + expect(result.verdict.kind).toBe("stale_test"); + if (result.verdict.kind === "stale_test") { + expect(result.verdict.candidate.specPath).toBe("tests/generated/cand.spec.ts"); + expect(result.verdict.fidelity.stable).toBe(true); + } + }); + + it("product_regression — the re-drive gives up, with the model's reason", async () => { + const learning = new InMemoryLearningStore(); + const result = await runTriage( + deps(new ScriptedRunner("failed"), drives({ finished: false, stopReason: "gave_up", gaveUpText: "no checkout button" }), learning), + OPTIONS, + ); + expect(result.verdict.kind).toBe("product_regression"); + if (result.verdict.kind === "product_regression") { + expect(result.verdict.reason).toContain("no checkout button"); + } + const recorded = await learning.priorFailures("REQ-A"); + expect(recorded.map((f) => f.reason).join()).toContain("[triage:product_regression]"); + }); + + it("product_regression — a finished but assertion-free re-drive proves nothing held", async () => { + const result = await runTriage( + deps(new ScriptedRunner("failed"), drives({ actions: [{ type: "goto", url: "http://x/" }] })), + OPTIONS, + ); + expect(result.verdict.kind).toBe("product_regression"); + if (result.verdict.kind === "product_regression") { + expect(result.verdict.reason).toContain("could not assert"); + } + }); + + it("inconclusive — the old spec cannot even run (runner error)", async () => { + const result = await runTriage(deps(new ScriptedRunner("throw"), drives({})), OPTIONS); + expect(result.verdict.kind).toBe("inconclusive"); + if (result.verdict.kind === "inconclusive") { + expect(result.verdict.reason).toContain("could not run the verifying spec"); + } + }); + + it("inconclusive — the re-drive itself fails to run (model outage)", async () => { + const learning = new InMemoryLearningStore(); + const result = await runTriage(deps(new ScriptedRunner("failed"), throwingDrive, learning), OPTIONS); + expect(result.verdict.kind).toBe("inconclusive"); + if (result.verdict.kind === "inconclusive") { + expect(result.verdict.reason).toContain("model call failed twice"); + } + // A drive that never ran records nothing — there is no attempt to learn from. + expect(await learning.priorFailures("REQ-A")).toHaveLength(0); + }); + + it("inconclusive — the fresh spec does not stabilize (flake, not proven regression)", async () => { + const learning = new InMemoryLearningStore(); + const result = await runTriage( + deps(new ScriptedRunner("failed", ["passed", "failed"]), drives({}), learning), + OPTIONS, + ); + expect(result.verdict.kind).toBe("inconclusive"); + if (result.verdict.kind === "inconclusive") { + expect(result.verdict.reason).toContain("did not stabilize"); + expect(result.verdict.reason).toContain("1/2"); + } + expect((await learning.priorFailures("REQ-A")).length).toBe(1); + }); + + it("inconclusive — the candidate never runs cleanly (runner errors on every attempt)", async () => { + const result = await runTriage( + deps(new ScriptedRunner("failed", ["throw", "throw"]), drives({})), + OPTIONS, + ); + expect(result.verdict.kind).toBe("inconclusive"); + if (result.verdict.kind === "inconclusive") { + expect(result.verdict.reason).toContain("never ran cleanly"); + } + }); + + it("records nothing to learning on a stale_test verdict", async () => { + const learning = new InMemoryLearningStore(); + await runTriage(deps(new ScriptedRunner("failed", ["passed", "passed"]), drives({}), learning), OPTIONS); + expect(await learning.priorFailures("REQ-A")).toHaveLength(0); + }); +}); + +describe("triage CLI contract", () => { + it("parses defaults and requirements", () => { + const args = parseTriageArgs(["--graph-file", "g.json", "--capability", "REQ-A", "--url", "http://x/"]); + expect(args).toMatchObject({ + graphFile: "g.json", + capability: "REQ-A", + url: "http://x/", + baseUrl: "http://x/", + n: 3, + outDir: "tests/generated", + allowShell: false, + json: false, + }); + expect(() => parseTriageArgs(["--url", "http://x/", "--capability", "R"])).toThrow(/--graph-file/); + expect(() => parseTriageArgs(["--graph-file", "g.json", "--url", "http://x/"])).toThrow(/--capability/); + expect(() => parseTriageArgs(["--graph-file", "g.json", "--capability", "R"])).toThrow(/--url/); + }); + + it("maps every verdict to its exit code — the code is the contract", async () => { + const cases: [Runner, TriageDeps["drive"], number][] = [ + [new ScriptedRunner("passed"), drives({}), 0], // not_reproducible + [new ScriptedRunner("failed", ["passed", "passed"]), drives({}), 0], // stale_test + [new ScriptedRunner("failed"), drives({ finished: false, stopReason: "gave_up" }), 1], // regression + [new ScriptedRunner("failed"), throwingDrive, 3], // inconclusive + ]; + for (const [runner, drive, code] of cases) { + const result = await runTriage(deps(runner, drive), OPTIONS); + expect(triageExitCode(result)).toBe(code); + } + }); + + it("renders the verdict for humans and emits the stable JSON contract", async () => { + const result = await runTriage(deps(new ScriptedRunner("failed", ["passed", "passed"]), drives({})), OPTIONS); + const human = renderTriageResult(result); + expect(human).toContain("Verdict: STALE TEST"); + expect(human).toContain("Repair candidate: tests/generated/cand.spec.ts"); + + const json = JSON.parse(triageToJson(result)) as Record; + expect(json["verdict"]).toBe("stale_test"); + expect(json["candidateSpec"]).toBe("tests/generated/cand.spec.ts"); + expect(json["oldSpec"]).toBe("tests/a.spec.ts"); + }); +});