Skip to content

Handle uneven PaperBench seed counts when parsing runs - #141

Closed
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/paperbench-uneven-seeds
Closed

Handle uneven PaperBench seed counts when parsing runs#141
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/paperbench-uneven-seeds

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Allow parse_run_data() to build incomplete PaperBench evaluation runs when papers have uneven numbers of available seeds instead of indexing past shorter histories.

The parser computes the maximum seed count across papers, but its short-paper guard currently skips only when seed == len(data). If one paper has one run and another has three, seed 2 satisfies 2 != 1 and the code attempts data[2], raising IndexError.

Fixes #140.

Fix

Skip whenever seed >= len(data).

Regression coverage

Adds a unit regression with one paper containing one run and another containing three runs. It verifies parsing produces three EvaluationRun objects, with the shorter paper present only in the first run and no out-of-range access.

Timestamp ordering, recent-seed selection, disqualification handling, and complete-run filtering are unchanged.

Copy link
Copy Markdown
Author

Superseded by #150. The run-group-based parser in #150 removes the positional per-paper indexing entirely, so uneven paper histories remain incomplete within their actual run group instead of causing out-of-range access or borrowing records from another run. #150 now explicitly fixes #140 as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PaperBench run parsing crashes on uneven seed counts

1 participant