QA-GUIDE.md §3 is the whole of what a producer is told about mutation validation:
For each new test, apply ONE targeted mutation to the line it claims to cover
(negate the guard, flip the comparison, drop the call), confirm the test fails,
restore. A test that survives its own mutation is decoration.
That is a complete, self-contained recipe, and it never names mutation-probe — the
bundled bin the adversarial-mutation-test skill actually runs at its step 3
(nix run github:rainlanguage/adversarial-mutation-test#mutation-probe -- mutants.toml;
--help is the manual). campaign-prompt.txt step 4 does say to INVOKE the skill rather
than hand-copy its rules, but §3 wins for anyone reading the guide alone — and §8's
mandatory Mutations applied evidence line sends every producer back to §3 to produce it.
The result is that agents hand-roll an edit-run-restore harness per run and write up the
matrix it produced as evidence.
The gap is not stylistic. A hand loop can ASSERT the four things the bin ENFORCES, and can
prove none of them:
- a green, non-empty baseline before any probe (a red or zero-test baseline aborts);
- proof from the suite's own tally that it RAN — a crash, compile error or timeout is
NO-RUN, never a pass, so a broken build cannot be written up as a killed mutant;
- a target that occurs EXACTLY once in its file, so a mutant that silently applied twice
or not at all is a HARNESS-ERROR rather than a verdict;
- a byte-exact restore, verified and re-checked pristine before the next mutant.
A hand-rolled matrix is therefore unfalsifiable in exactly the way QA-GUIDE §2 says a green
suite is: it reports what the agent believes happened, not what ran.
campaign-prompt.txt step 3c(ii) carries the same recipe in miniature — "mutation-validate
per step 4 (break the covered line → the assertion FAILS → restore → PASSES)" — so the red-PR
path teaches the hand loop too.
Fix
- §3 routes to the skill and its bundled bin, gives the runnable command, sends file format /
verdicts / exit codes to --help rather than restating a snapshot of them, states the four
integrity properties as the REASON the tool is the route, and names the two duties the bin
cannot discharge (commit before the first probe; keep targets out of test code).
- The hand-roll goes, said as a prohibition and not merely omitted — a rule merely deleted is
a rule the next edit reinvents.
- §8's
Mutations applied line says the content is TRANSCRIBED from the probe's verdicts.
- Step 3c(ii)'s parenthetical points at the same probe.
- Pinned by a test, the way
the_producer_prompt_routes_screenshots_to_this_subcommand pins
the render-component retirement.
QA-GUIDE.md§3 is the whole of what a producer is told about mutation validation:That is a complete, self-contained recipe, and it never names
mutation-probe— thebundled bin the
adversarial-mutation-testskill actually runs at its step 3(
nix run github:rainlanguage/adversarial-mutation-test#mutation-probe -- mutants.toml;--helpis the manual).campaign-prompt.txtstep 4 does say to INVOKE the skill ratherthan hand-copy its rules, but §3 wins for anyone reading the guide alone — and §8's
mandatory
Mutations appliedevidence line sends every producer back to §3 to produce it.The result is that agents hand-roll an edit-run-restore harness per run and write up the
matrix it produced as evidence.
The gap is not stylistic. A hand loop can ASSERT the four things the bin ENFORCES, and can
prove none of them:
NO-RUN, never a pass, so a broken build cannot be written up as a killed mutant;or not at all is a
HARNESS-ERRORrather than a verdict;A hand-rolled matrix is therefore unfalsifiable in exactly the way QA-GUIDE §2 says a green
suite is: it reports what the agent believes happened, not what ran.
campaign-prompt.txtstep 3c(ii) carries the same recipe in miniature — "mutation-validateper step 4 (break the covered line → the assertion FAILS → restore → PASSES)" — so the red-PR
path teaches the hand loop too.
Fix
verdicts / exit codes to
--helprather than restating a snapshot of them, states the fourintegrity properties as the REASON the tool is the route, and names the two duties the bin
cannot discharge (commit before the first probe; keep targets out of test code).
a rule the next edit reinvents.
Mutations appliedline says the content is TRANSCRIBED from the probe's verdicts.the_producer_prompt_routes_screenshots_to_this_subcommandpinsthe render-component retirement.