Summary
paperbench.metrics.parse_run_data() builds evaluation runs across papers with different numbers of available seeds. It correctly identifies the maximum seed count, but its short-paper guard only checks equality:
if seed == len(data):
continue
paper_eval = data[seed]["paper_eval"]If one paper has 1 run and another has 3, seed 1 is skipped for the shorter paper, but seed 2 satisfies 2 != 1 and the code attempts data[2], raising IndexError.
Impact
Parsing historical/incomplete PaperBench run data can crash whenever per-paper run counts differ by more than one, instead of producing the intentionally incomplete EvaluationRun objects described by the surrounding comment.
Proposed resolution
Skip whenever the current seed index is outside that paper's available data:
if seed >= len(data):
continueAdd a regression with one paper containing one run and another containing three runs, verifying parsing produces three evaluation runs without indexing past the shorter paper.
Summary
paperbench.metrics.parse_run_data()builds evaluation runs across papers with different numbers of available seeds. It correctly identifies the maximum seed count, but its short-paper guard only checks equality:If one paper has 1 run and another has 3, seed 1 is skipped for the shorter paper, but seed 2 satisfies
2 != 1and the code attemptsdata[2], raisingIndexError.Impact
Parsing historical/incomplete PaperBench run data can crash whenever per-paper run counts differ by more than one, instead of producing the intentionally incomplete
EvaluationRunobjects described by the surrounding comment.Proposed resolution
Skip whenever the current seed index is outside that paper's available data:
Add a regression with one paper containing one run and another containing three runs, verifying parsing produces three evaluation runs without indexing past the shorter paper.