faultbank

A fault-injection recall bench for snag. Small, clean subjects with specific, known failure modes planted into them. We run snag map (or a blind mapping agent) against the injected variant with no hint of where the fault is, and measure recall — did the map name the mode we planted?

Backs EXP 0002 (the fault-injection recall gate).

Note — this bench is now public

It was originally held out and kept private, because the whole point of a recall bench is that the mapper is blind to the planted faults. snag is no longer an active product, so there's no live recall score left to protect, and this is published as the reproducibility artifact behind EXP 0002 — the actual subjects and answer key behind the reported result, so anyone can re-run it.

Because it is public, treat it as contaminated for any future blind measurement: a crawled or forked test set can leak into training data and can no longer be trusted as held-out. If you want a live blind bench, plant fresh faults in new subjects rather than reusing these.

To reproduce a blind pass yourself:

• Never feed SPEC.md (the answer key) or the clean/ reference to the mapper. Only the injected/ variant gets mapped.
• Stage the injected file at a neutral path with a neutral name (e.g. copy confirmed-runner/ts/injected/job_runner.ts → a scratch job_runner.ts) so the path itself (injected/) doesn't tip the model off.

Layout

confirmed-runner/          # scenario 1: a batch job runner (see SPEC.md)
  SPEC.md                  # the scenario + the 4 planted faults + expected trigger/manifestation  ← ANSWER KEY
  ts/  clean/ injected/    # TypeScript  (job_runner.ts)
  py/  clean/ injected/    # Python      (job_runner.py)
  go/  clean/ injected/    # Go          (job_runner.go)

Each language implements the same scenario and hosts the same four language-universal faults, idiomatically. clean/ is the reference (correct) implementation; injected/ is the same program with all four faults planted. The clean/injected pair differ only at the four fault sites.

Faults (same four across every language)

#	Class	Planted fault
F1	liveness (seed)	wait-for-confirmation has no timeout — blocks forever if upstream never answers
F2	observability	error path swallows the failure — job silently dropped, batch reports success
F3	io / atomicity	result write is in-place (no temp+rename) — crash mid-write corrupts output
F4	resource / concurrency	unbounded task-per-job — large batches exhaust resources

Full trigger/manifestation expectations are in confirmed-runner/SPEC.md.

Scoring (summary)

A planted fault counts as recalled if the generated map contains a node whose trigger/manifestation matches the SPEC.md expectation at reconcile-match rigor (not loose keyword overlap). Recall = recalled / planted, tallied per-fault, per-language, and per-class.

See the EXP 0002 write-up for thresholds and the run protocol.