Skip to content

QA-GUIDE §3 teaches a hand-rolled mutation loop and never names mutation-probe, so producers build their own harness and their evidence is unfalsifiable #336

Description

@thedavidmeister

QA-GUIDE.md §3 is the whole of what a producer is told about mutation validation:

For each new test, apply ONE targeted mutation to the line it claims to cover
(negate the guard, flip the comparison, drop the call), confirm the test fails,
restore. A test that survives its own mutation is decoration.

That is a complete, self-contained recipe, and it never names mutation-probe — the
bundled bin the adversarial-mutation-test skill actually runs at its step 3
(nix run github:rainlanguage/adversarial-mutation-test#mutation-probe -- mutants.toml;
--help is the manual). campaign-prompt.txt step 4 does say to INVOKE the skill rather
than hand-copy its rules, but §3 wins for anyone reading the guide alone — and §8's
mandatory Mutations applied evidence line sends every producer back to §3 to produce it.
The result is that agents hand-roll an edit-run-restore harness per run and write up the
matrix it produced as evidence.

The gap is not stylistic. A hand loop can ASSERT the four things the bin ENFORCES, and can
prove none of them:

  • a green, non-empty baseline before any probe (a red or zero-test baseline aborts);
  • proof from the suite's own tally that it RAN — a crash, compile error or timeout is
    NO-RUN, never a pass, so a broken build cannot be written up as a killed mutant;
  • a target that occurs EXACTLY once in its file, so a mutant that silently applied twice
    or not at all is a HARNESS-ERROR rather than a verdict;
  • a byte-exact restore, verified and re-checked pristine before the next mutant.

A hand-rolled matrix is therefore unfalsifiable in exactly the way QA-GUIDE §2 says a green
suite is: it reports what the agent believes happened, not what ran.

campaign-prompt.txt step 3c(ii) carries the same recipe in miniature — "mutation-validate
per step 4 (break the covered line → the assertion FAILS → restore → PASSES)" — so the red-PR
path teaches the hand loop too.

Fix

  • §3 routes to the skill and its bundled bin, gives the runnable command, sends file format /
    verdicts / exit codes to --help rather than restating a snapshot of them, states the four
    integrity properties as the REASON the tool is the route, and names the two duties the bin
    cannot discharge (commit before the first probe; keep targets out of test code).
  • The hand-roll goes, said as a prohibition and not merely omitted — a rule merely deleted is
    a rule the next edit reinvents.
  • §8's Mutations applied line says the content is TRANSCRIBED from the probe's verdicts.
  • Step 3c(ii)'s parenthetical points at the same probe.
  • Pinned by a test, the way the_producer_prompt_routes_screenshots_to_this_subcommand pins
    the render-component retirement.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions