Skip to content

The mutation ledger predates almost the entire current source tree #64

Description

@thedavidmeister

Audit finding cov-12 — dimension 2, severity LOW. Whole-repo audit pass 1 at 440e90b5.

audit/mutation-test-scans.json:1-22

Problem

The only ledgered adversarial-mutation campaign is at commit 4422e291 (2026-07-25, tag sol-v0.1.4 +2, scope "whole repo", 91 behaviours). The repo is at 440e90b5. git diff --stat 4422e29..HEAD is +9219/-431 across 68 files, and every one of these source files was ADDED after that campaign, so no behaviour in them has ever been mutation-proven:

  • src/abstract/RainDeployBroadcast.sol, RainDeploySuitesBase.sol, RainDeployVerifyBase.sol, RainDeployVerifyChain.sol, RainDeployVerifySnapshot.sol, RegistryDeploySuites.sol (all new)
  • src/concrete/AddressRegistry.sol, src/concrete/MigrationRegistry.sol — both deployed contracts (new)
  • src/interface/IAddressRegistryV1.sol, IMigrationRegistryV1.sol (new)
  • src/lib/LibAddressRegistry.sol, LibMigrationRegistry.sol, LibRainDeploySnapshot.sol (772 lines, new)
  • script/Build.sol, script/Deploy.sol (new)
  • src/lib/LibRainDeploy.sol grew by 193 lines (checkResolvedAddresses*, the pre-fork address derivation)

The ledgered coverage therefore attests to roughly a fifth of the current repo. The gaps in cov-01 through cov-11 are the ones a read found; a campaign is what says whether the remainder of these ~9k lines is covered by tests that actually kill mutants, and several of the gaps above (cov-07's dependency loop, cov-09's != 0x20, cov-06's outer loop) are exactly the class a mutation pass finds mechanically and a read finds by luck.

Proposed fix

Re-run the campaign at the current commit and re-ledger, rather than adding a test:

# scope: whole repo, at 440e90b5ce0875871dfcb37fbee680a23494f08f
nix develop -c forge test   # baseline must be green before mutating
# then drive the adversarial-mutation-test skill at whole-repo scope

and append a second entry to audit/mutation-test-scans.json in the existing shape:

  {
    "timestamp": "<ISO8601>",
    "commit": "440e90b5ce0875871dfcb37fbee680a23494f08f",
    "publishedTag": "sol-v0.1.5",
    "commitsAheadOfTag": <n>,
    "scope": "whole repo",
    "tool": "adversarial-mutation-test",
    "skillVersion": "<version>",
    "summary": { "units": <n>, "behaviours": <n>, "killedByExistingTests": <n>, "gapsFilled": <n>, "equivalentMutants": <n>, "unkillableDefensiveGuards": <n>, "candidates": <n>, "confirmed": <n>, "filed": [] }
  }

Prioritise the units with no ledgered coverage and no test file of their own: script/Build.sol (cov-01), LibRainDeploySnapshot.freeze/writeAliasLib (cov-02, cov-03, cov-04), and RainDeployBroadcast.run (cov-05).


Verification — this finding survived an adversarial refutation pass

Could not refute; every claim checks out against the source. audit/mutation-test-scans.json contains exactly one entry (commit 4422e29, scope "whole repo", 1 unit / 91 behaviours), and 4422e29 is an ancestor of HEAD 440e90b. git ls-tree -r 4422e291 restricted to src/, script/, test/ returns only four files (src/lib/LibRainDeploy.sol plus three test files), so every source file the finding enumerates — src/concrete/AddressRegistry.sol, src/concrete/MigrationRegistry.sol, all six src/abstract/RainDeploy*/RegistryDeploySuites.sol, both interfaces, LibAddressRegistry/LibMigrationRegistry/LibRainDeploySnapshot (772 lines), script/Build.sol, script/Deploy.sol — was added after the ledgered campaign, and LibRainDeploy.sol itself gained 193 lines including checkResolvedAddresses/checkResolvedAddressesOnNetworks. git diff --stat 4422e29..HEAD is exactly 68 files, +9219/-431 as claimed. git log --all --grep=mutation -i shows only the 2026-07-25 campaign (PR #19); PRs #20-#43 landed the whole deployed surface with no ledgered or commit-evidenced mutation pass, against the org's "adversarial mutation pass on every code PR" convention. No exempting convention exists: grep over all md/json/yaml finds no documentation of the ledger beyond the file itself, CLAUDE.md never scopes it to release time, .gitignore has no campaign scratch path, and CI (rainix-sol-legal/test/static) runs no mutation gate. The existence of test files does not refute it — the claim is about attestation of mutant-killing, not about tests existing. One inaccuracy inside the proposed fix, not fatal to the finding: it prioritises units with "no test file of their own" and names RainDeployBroadcast.run and LibRainDeploySnapshot.freeze, but test/src/abstract/RainDeployBroadcast.t.sol exists (testRunSelectsTheSuiteFromTheEnvBeforeTheKeyAndNeverDefaults) and test/src/lib/LibRainDeploySnapshot.t.sol covers freeze with two tests; only script/Build.sol and writeAliasLib genuinely lack a direct test, so the prioritisation list should be trimmed. Severity LOW is correct by value-at-risk: CLAUDE.md states nothing is released yet, so what is at risk is the unproven quality of tests over code destined for deployment, not a live on-chain defect.

Metadata

Metadata

Labels

auditAudit findingpass1Audit pass 1 (whole-repo, 2026-08-15)severity:lowAudit severity: LOW

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions