PaperBench metrics can mix papers from different evaluation runs into one seed

[sylvesterkaczmarek]

Summary

paperbench.metrics.parse_run_data() groups each paper's records independently by recency and then combines records with the same ordinal position into an EvaluationRun.

That loses the actual run_group_id identity of the evaluation run. If one run is missing a paper, the per-paper lists shift and a synthetic seed can combine papers from different real runs.

Example

Suppose an agent has two run groups, newest run_B and older run_A:

• paper 1: run_B, run_A
• paper 2: only run_A

The current code sorts each paper independently and constructs seed 0 from the first entry of each list. Seed 0 therefore combines paper 1 from run_B with paper 2 from run_A, making an incomplete run appear more complete and mixing scores across actual evaluation runs.

Impact

compute_agg_stats() filters for complete runs. Mis-grouping can therefore change which runs are treated as complete and alter reported PaperBench aggregate scores.

Proposed resolution

Preserve run_group_id while parsing and construct each EvaluationRun from records sharing the same actual run group. Rank whole run groups by timestamp when applying seeds_to_keep, rather than ranking each paper independently.

Add regression coverage with a missing paper in the newest run so records from the older run cannot be pulled into it.