What happens
Any pull request that re-records the eval corpus gets its cassettes reviewed as
though they were hand-authored source. On #115 that produced 62 false-positive
findings across 62 threads, and failed three of tinysweeper's own lane checks
(critique, security, description). Five findings on the actual source
change were buried underneath.
Every one of the 62 was wrong, in one of two ways.
"Add the required resolved / late field" — 42 threads. Both fields are
#[serde(default)] (src/harness/schema.rs), so an absent field is valid and
deserializes to empty/false. Nothing is missing and nothing fails to parse.
"This cassette is about the wrong pull request" — 20 threads, variously
phrased as Move cassette to the correct case directory, Use the kernel-bypass
PR's cassette content, not the summarise PR's, Align cassette summary with the
eval case's PR. These read the case id as a description of the fixture.
ts-0045-kernel-bypass-hallucination is named for the hallucination it guards
against — its title is "A loaded phrase in a commit subject must not become a
security finding" — and its fixture is tinysweeper#45, a change to
src/findings/summarise.rs. Cassette content about summarise is correct;
content about kernel bypass would be the bug.
Why it is a category error, not a tuning problem
A cassette is not a fixture anyone writes. It is a verbatim recording of what a
model returned, produced by tinysweeper eval run --record. Reviewing one asks
the lanes to critique a model's past output as if the author had chosen every
byte, and the only correct response to any finding on a cassette is "re-record
it" — never "edit it". Acting on one of these suggestions would make the replay
assert against a response that never happened, which is precisely what the
corpus exists to prevent.
Suggested fix
[paths].ignore in src/config/defaults.toml already exists for this and
already covers the closest analogue, **/*.snap:
[paths]
ignore = [
"**/*.lock",
"vendor/**",
"**/generated/**",
"**/node_modules/**",
"**/*.min.js",
"**/*.snap",
]
Adding evals/cassettes/** would fix it. Worth considering evals/runs/** at
the same time, and whether the list should carry a comment saying what unites
its entries — every one is a generated artifact, and cassettes were omitted
because that rule was implicit rather than written down.
Evidence
What happens
Any pull request that re-records the eval corpus gets its cassettes reviewed as
though they were hand-authored source. On #115 that produced 62 false-positive
findings across 62 threads, and failed three of tinysweeper's own lane checks
(
critique,security,description). Five findings on the actual sourcechange were buried underneath.
Every one of the 62 was wrong, in one of two ways.
"Add the required
resolved/latefield" — 42 threads. Both fields are#[serde(default)](src/harness/schema.rs), so an absent field is valid anddeserializes to empty/false. Nothing is missing and nothing fails to parse.
"This cassette is about the wrong pull request" — 20 threads, variously
phrased as Move cassette to the correct case directory, Use the kernel-bypass
PR's cassette content, not the summarise PR's, Align cassette summary with the
eval case's PR. These read the case id as a description of the fixture.
ts-0045-kernel-bypass-hallucinationis named for the hallucination it guardsagainst — its title is "A loaded phrase in a commit subject must not become a
security finding" — and its fixture is tinysweeper#45, a change to
src/findings/summarise.rs. Cassette content aboutsummariseis correct;content about kernel bypass would be the bug.
Why it is a category error, not a tuning problem
A cassette is not a fixture anyone writes. It is a verbatim recording of what a
model returned, produced by
tinysweeper eval run --record. Reviewing one asksthe lanes to critique a model's past output as if the author had chosen every
byte, and the only correct response to any finding on a cassette is "re-record
it" — never "edit it". Acting on one of these suggestions would make the replay
assert against a response that never happened, which is precisely what the
corpus exists to prevent.
Suggested fix
[paths].ignoreinsrc/config/defaults.tomlalready exists for this andalready covers the closest analogue,
**/*.snap:Adding
evals/cassettes/**would fix it. Worth consideringevals/runs/**atthe same time, and whether the list should carry a comment saying what unites
its entries — every one is a generated artifact, and cassettes were omitted
because that rule was implicit rather than written down.
Evidence
Stop a finding's severity drifting between pushes #115
resolved/latedefaults:src/harness/schema.rsevals/cases/ts-0045-kernel-bypass-hallucination.toml