Fix runner resume behaviour, --clean flag, and add --seed

[dan-s-w]

Three independent problems in eval_runner.py, all reachable from a normal resumed run. Split into one PR because they're the same file and one reviewer will read them together.

1. Per-sampler filtering narrowed the shared dataset

run_evals reassigned dataset.df to the current sampler's remaining problems ([old line 129]). The next sampler then computed its remaining set from that already-narrowed frame.

On a resumed run, every sampler after the first silently received fewer problems than the dataset contains — so accuracy was computed over a smaller, arbitrarily chosen subset with nothing in the output indicating it. Filtering now reads from a per-dataset snapshot and leaves dataset.df untouched.

Worth noting DATASETS holds mutable dataclass instances at module scope, so the narrowing persisted for the life of the process.

2. FAILED rows were treated as completed

get_remaining_problems matched on the query column regardless of outcome, so a row written as FAILED was never retried. A transient timeout or rate limit became a permanent hole in that sampler's results — and since write_metrics drops FAILED rows from both numerator and denominator, that hole is invisible in the scorecard.

FAILED rows are now considered outstanding and retried on the next run.

A retried-and-succeeded query leaves both the old FAILED row and the new successful row in the CSV. That's harmless for the current metrics (FAILED rows are excluded), but flagging it in case you'd prefer a dedupe on write.

3. --clean took a string

parser.add_argument("--clean", default=False, type=str, ...)
...
if args.clean:

--clean False passes the non-empty string "False", which is truthy — so it wiped the results folder, the exact opposite of what it reads like. Now action="store_true".

Also: --seed

Applied to the --limit sample so two limited runs can use the same subset. Default unchanged (unseeded), so existing behaviour is untouched.

Tests

New tests/test_eval_runner.py — stub sampler, no network, runs without API keys.

All three regressions were confirmed to fail against the previous implementation:

FAILED test_failed_problems_are_retried
FAILED test_explicit_problems_argument_is_not_the_shared_dataframe
FAILED test_second_sampler_is_not_narrowed_by_the_first
3 failed, 1 passed

and pass after the change.

🤖 Generated with Claude Code