Round-2 adversarial-mutation-test coverage: query, magic CLI, validators, CopyArtifacts

[thedavidmeister]

Round-2 AMT campaign over the units left uncovered by round 1: process_meta_query/get_authoring_meta (crates/cli/src/meta/query/mod.rs), the magic ls CLI surface, the common/v1 validator regexes, and script/CopyArtifacts.sol — probed with nix run github:rainlanguage/adversarial-mutation-test#mutation-probe over two suites: rust nix develop rainix#rust-shell -c cargo test (baseline 137 passed) and sol git checkout -- crates/bindings/abi && git clean -fdq crates/bindings/abi && forge build && forge script ./script/CopyArtifacts.sol --ffi && forge test (baseline 16 passed).

Sequence per the AMT protocol: all 49 mutants (40 rust + 9 sol) probed against the PRE-EXISTING suites first — rust 7 KILLED / 33 SURVIVED / 0 no-run, sol 4 KILLED / 4 SURVIVED / 1 NO-RUN (compile-enforced) — then every survivor either killed by a new or in-place-strengthened discriminating test and re-probed (--only pass 2: rust 21/21 KILLED, sol S1–S3 3/3 KILLED, both baselines green at 146/16), or recorded below as equivalent/dead/build-enforced with evidence, never forced.

Behaviour matrix

Verdict legend: PRE = killed by pre-existing test (validates existing coverage), NEW = killed by a test added here, STR = killed by a pre-existing test strengthened in place.

#	Behaviour	Mutation(s)	Verdict	Killing test(s)
1	process_meta_query: absent data member → NoRecordFound	R01	NEW	meta::query::tests::test_process_meta_query_missing_data_is_no_record_found
2	process_meta_query: null meta (true absence) → NoRecordFound	R02	NEW	test_process_meta_query_missing_meta_is_no_record_found
3	process_meta_query: undecodable rawBytes hex → NoRecordFound, not CorruptMeta, not swallowed to empty	R03, R04	NEW	test_process_meta_query_bad_hex_is_no_record_found
4	process_meta_query: POST with the JSON query body to the given url	R05, R06	NEW	test_process_meta_query_success_exact_bytes, test_process_meta_query_non_json_is_reqwest_error
5	process_meta_query: transport error → ReqwestError	R07	NEW	test_process_meta_query_send_error_is_reqwest_error
6	process_meta_query: body decode error → ReqwestError	R08	NEW	test_process_meta_query_non_json_is_reqwest_error
7	process_meta_query: returned bytes are exactly the decoded rawBytes	R09	NEW	test_process_meta_query_success_exact_bytes
8	get_authoring_meta: unpack failure skips the item and keeps scanning	A01	NEW	test_get_authoring_meta_skips_unpack_failure
9	get_authoring_meta: scan continues beyond the first item	A02	NEW	test_get_authoring_meta_scans_beyond_first_item
10	magic ls prints every KnownMagic, in order, correct columns, 0x-prefixed exact values	G01–G05	NEW	tests/magic_cli.rs::test_magic_ls_exact_output (e2e via CARGO_BIN_EXE_rain-metadata)
11	RainSymbol: lowercase-only tail; end anchor	C11 (tail widened to uppercase)	STR	test_rain_symbol_validate (added "aA" invalid)
		C12 (end anchor dropped)	PRE	test_rain_symbol_validate
12	SolidityIdentifier: end anchor; no hyphen in tail	C13, C14	PRE	test_solidity_identifier_validate
13	RainString: excludes DEL; whitespace class not narrowed to space	C15 (DEL)	STR	test_rain_string_validate (added "\u{7f}" invalid)
		C22 (whitespace narrowed)	PRE	test_rain_string_validate
14	RainTitle: first char printable non-space, excludes DEL; single-char alternative	C16 (DEL first char)	STR	test_rain_title_validate (added DEL-leading invalid)
		C17 (single-char alternative dropped)	PRE	test_rain_title_validate
15	HASH_PATTERN: exactly 64 chars; hex class; lowercase 0x prefix	C18, C19 (63/65 length)	PRE	test_hash_pattern
		C20 (class widened to g), C21 (0X accepted)	STR	test_hash_pattern (added 64×g and "0X"+64-hex invalids)
16	CopyArtifacts.run(): every contract copied; loop body does the copy; content byte-exact	S1 (first contract only), S2 (body no-op)	STR	testArtifactsCommitted (corrupt both committed files → run() → byte-exact restore against pre-corruption on-disk state)
17	CopyArtifacts.run(): missing dst is recreated, exists guard not inverted	S3	STR	testArtifactsCommitted (delete one committed file → run() → recreated byte-exact)
18	CopyArtifacts: writeFile content/target-path/stable-subset/presence	S5, S6, S7, S8	PRE	testArtifactsCommitted
19	IMetaBoardV1_2.emitMeta interface name	S9	NO-RUN	compile-enforced: rename fails forge build at LibDescribedByMeta.sol:37

Equivalent / dead / build-enforced dispositions (no test, never forced)

• L01 equivalent — serde_cbor 0.11.2 SliceRead raises every EOF at offset == slice.len(), and cbor_decode compares against exactly that slice length, so == vs <= is unobservable. Empirical truncation sweep (>10 EOF cases, all offset == len).
• L02 equivalent — track/metas are pushed in the same iteration; the length check can never fire.
• L03 equivalent — dotrain create() uri-gate half removal unobservable.
• L04/L05 dead branches — panic-armed mutants survive, proving the branches unreached.
• P03/P06/P07 equivalent — ValidationErrors::merge_all drops all child results (children are keyed by their own field names / Field kind and never pass the err.remove(field) + Struct filter; Operand carries zero field rules). Behaviour already on file as BitIntegerRange/OperandArgRange/Output validation drops all sub-validation errors (ValidationErrors::merge_all misuse) #173/OpMeta validation silently discards nested BitInteger/Output errors (merge_all misused) #214 — no new issue.
• V02/V03 equivalent — zero #[validate] attrs in solidity_abi/v2.rs make the validate gates vacuous; already on file as SolidityAbiMeta validation is vacuous: Validate derives carry no rules, so the TryFrom validate gate and per-index annotation can never reject #153.
• MB03 equivalent — cynic rejects data-absent + errors-absent responses ("Either data or errors must be present"), so the Empty arm is unreachable; already on file as CynicClientError::Empty is unreachable from CynicClient::query: missing/null data surfaces as a Request decode error #178. Empirical: {"data":null}, {}, {"data":null,"errors":null} all surface as Request decode errors.
• B01 pass-1 verdict unsound (stale artifact); actually build-enforced — with a clean OUT_DIR, mutating the metaboard build.rs schema registration name fails compilation at lib.rs:5 #[cynic::schema("metaboard")] ("Could not find a schema with this name"), i.e. NO-RUN class like S9. The pass-1 SURVIVED verdict came from the probe's incremental build reusing stale OUT_DIR cynic-schemas/metaboard.{graphql,rs} residue from the unmutated baseline; demonstrated both ways (clean OUT_DIR → exit 101; stale artifacts restored → mutated crate builds green). No runtime test can pin a build-registry name.
• S4 equivalent — vm.writeFile truncates before writing, so the remove-before-write is redundant for an existing dst.
• U1-M03 (no syntactic mutant) — #[serde] Option fields decode absent keys to None identically with or without an explicit default; analytic serde equivalence recorded.

Adjudication A-1: disproof

The round-1 adjudication claim that generate source emits empty meta was DISPROVEN: fresh cargo build at bba50a7, printf '#main\n_: int-add(1 2);' | rain-metadata generate source yields selector 0x37480e2a, meta length 0x46, calldata meta bytes identical to the independently CBOR-encoded document, ff0a89c674ee7874 document prefix present. The "empty meta" observation was a stale-binary artifact.

Deviation: forge runs same-contract tests in parallel

The plan assumed test functions within one contract are serialized; forge 1.7.2 disproved this (new functions appended to CopyArtifactsTest raced testArtifactsCommitted on the shared on-disk committed files, flaking the suite). Since every step shares the same files, testArtifactsCommitted was instead strengthened IN PLACE into one serialized sequence: original assertions first (unweakened) → corrupt both committed files → run() → assert byte-exact restore (kills S1/S2) → delete one committed file → run() → assert recreated byte-exact (kills S3).

Issues filed (adversarial findings, labels audit+adversarial)

#231 (solc artifact prints null, exit 0, for a missing component — adjudication A-2), #232 (Store::merge deployer_hash_map overwrite asymmetry — A-3), #233 (str_to_bytes32/bytes32_to_str NUL round-trip asymmetry — A-4), #234 (process_meta_query collapses corrupt payloads / bad hex / GraphQL errors into NoRecordFound; meta-side sibling of #213), #235 (RainSymbol regex accepts trailing/double hyphens vs kebab-case contract), #236 (RainTitle validator message contradicts its regex and doc). Duplicates deliberately not re-filed: merge_all child-dropping (#173/#214), cynic Empty unreachability (#178), vacuous SolidityAbiMeta validation (#153).

QA

• Discriminating tests: meta::query::tests::* (6 httpmock process_meta_query tests + 2 multi-item get_authoring_meta tests), tests/magic_cli.rs::test_magic_ls_exact_output (e2e against the built binary), strengthened in place test_rain_symbol_validate / test_rain_string_validate / test_rain_title_validate / test_hash_pattern (new discriminating invalids only — no existing case weakened), sol testArtifactsCommitted (corrupt-both-restore + delete-one-recreate appended after the original unweakened assertions). Every pass-1 survivor selected for pass 2 flips to KILLED with the killer named by the probe fail-pattern (probe-rust-pass2: 21/21; probe-sol-pass2: 3/3; baselines green 146 / 16).
• Mutations applied: 49 exact-string mutants — 40 rust (R01–R09, A01–A02, G01–G05, C11–C22, L01–L05, P03/P06/P07, V02/V03, MB03, B01) + 9 sol (S1–S9); per-mutant line → mutation → verdict in the matrix above (e.g. .ok_or(Error::NoRecordFound) → CorruptMeta → test_process_meta_query_missing_data_is_no_record_found; CopyArtifacts exists-guard inverted → delete-one-recreate sequence).
• Oracle: doc comments of the mutated units (process_meta_query's "resolving if a record was found" contract, common/v1 regex doc contracts, CopyArtifacts committed-artifact invariant), the KnownMagic table for the exact magic ls pin, httpmock request-shape assertions (method, path, body), and independently constructed expected values (expected meta bytes hex-encoded in the test, not read back from the implementation; sol byte-exact oracle = pre-corruption on-disk state captured before mutating the files).
• Category check: round-2 worklist covers 49 mutants across 4 unit groups; all 49 dispositioned — rust 28 KILLED (7 PRE + 21 NEW/STR) + 12 equivalent/dead/build-enforced, sol 7 KILLED (4 PRE + 3 STR) + 1 equivalent (S4) + 1 compile-enforced (S9); adversarial candidates filed as CLI solc artifact prints null with exit 0 when the requested component is missing from the artifact #231–RainTitle validator message claims 'alphanumeric ASCII letters and spaces' but the regex accepts all printable ASCII and rejects edge spaces #236.

Probe tool: nix run github:rainlanguage/adversarial-mutation-test#mutation-probe with the two suite commands above, proof (\d+) passed; (\d+) failed, fail-pattern naming killer tests. Mutant files kept out of the repo.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

@thedavidmeister, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 55 minutes

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 5957053b-79de-4ace-8956-475997f58d88

📥 Commits

Reviewing files that changed from the base of the PR and between bba50a7 and 0ab172b.

📒 Files selected for processing (4)

• crates/cli/src/meta/query/mod.rs
• crates/cli/src/meta/types/common/v1.rs
• crates/cli/tests/magic_cli.rs
• test/script/CopyArtifacts.t.sol

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 2026-08-22-amt-round2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}