Preserve PaperBench run group identity when aggregating seeds - #150
Preserve PaperBench run group identity when aggregating seeds#150sylvesterkaczmarek wants to merge 7 commits into
Conversation
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b9e90f7c86
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1430042a37
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
1 participant
Summary
Keep PaperBench paper results grouped by the actual nanoeval run identity instead of aligning each paper independently by recency rank.
The previous parser sorted runs separately for every paper and then combined entries at the same list index into one synthetic seed. If a newer run was missing a paper, that paper could be pulled from an older run, mixing two real evaluation runs and potentially making an incomplete run appear complete. Grouping only by
run_group_idalso collapses distinctn_triesattempts because nanoeval places every attempt from one evaluation under the same run group.Fixes #140.
Fixes #149.
Fix
run_group_idin parsed entries;group_id=<attempt_id>.<retry_idx>;(run_group_id, attempt_id)when attempt metadata is available;run_group_id;seeds_to_keep;Regression coverage
Adds deterministic coverage for incomplete newer runs, whole-run
seeds_to_keep, nanoeval attempt/retry identity parsing, and multiple attempts sharing the samerun_group_id. Distinct attempts remain separateEvaluationRunobjects instead of overwriting one another.Scoring, disqualification handling, per-paper scores, and aggregate-stat formulas are unchanged.