Skip to content

Lanes review recorded eval cassettes as source, producing large volumes of false positives #116

Description

@senamakel

What happens

Any pull request that re-records the eval corpus gets its cassettes reviewed as
though they were hand-authored source. On #115 that produced 62 false-positive
findings across 62 threads
, and failed three of tinysweeper's own lane checks
(critique, security, description). Five findings on the actual source
change were buried underneath.

Every one of the 62 was wrong, in one of two ways.

"Add the required resolved / late field" — 42 threads. Both fields are
#[serde(default)] (src/harness/schema.rs), so an absent field is valid and
deserializes to empty/false. Nothing is missing and nothing fails to parse.

"This cassette is about the wrong pull request" — 20 threads, variously
phrased as Move cassette to the correct case directory, Use the kernel-bypass
PR's cassette content, not the summarise PR's
, Align cassette summary with the
eval case's PR
. These read the case id as a description of the fixture.
ts-0045-kernel-bypass-hallucination is named for the hallucination it guards
against — its title is "A loaded phrase in a commit subject must not become a
security finding" — and its fixture is tinysweeper#45, a change to
src/findings/summarise.rs. Cassette content about summarise is correct;
content about kernel bypass would be the bug.

Why it is a category error, not a tuning problem

A cassette is not a fixture anyone writes. It is a verbatim recording of what a
model returned, produced by tinysweeper eval run --record. Reviewing one asks
the lanes to critique a model's past output as if the author had chosen every
byte, and the only correct response to any finding on a cassette is "re-record
it" — never "edit it". Acting on one of these suggestions would make the replay
assert against a response that never happened, which is precisely what the
corpus exists to prevent.

Suggested fix

[paths].ignore in src/config/defaults.toml already exists for this and
already covers the closest analogue, **/*.snap:

[paths]
ignore = [
    "**/*.lock",
    "vendor/**",
    "**/generated/**",
    "**/node_modules/**",
    "**/*.min.js",
    "**/*.snap",
]

Adding evals/cassettes/** would fix it. Worth considering evals/runs/** at
the same time, and whether the list should carry a comment saying what unites
its entries — every one is a generated artifact, and cassettes were omitted
because that rule was implicit rather than written down.

Evidence

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority: p1Next. Wrong behaviour a user will hit, or a security weakness behind a condition.

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions