Skip to content

fix(paperbench): preserve solver shortnames in metrics parsing - #174

Open
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/paperbench-agent-id-parsing
Open

fix(paperbench): preserve solver shortnames in metrics parsing#174
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/paperbench-agent-id-parsing

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Preserve complete PaperBench solver shortnames when reconstructing agent IDs from run-group records.

PaperBench creates run-group IDs as:

<timestamp>_run-group_<solver.shortname()>

but parse_run_data() currently derives the agent ID with:

run_group_id.split("_")[-1]

That truncates any solver shortname containing underscores. The built-in direct-submission solver reports direct_submission_solver, so its records are currently grouped under the incorrect agent ID solver.

Fix

Parse the generated _run-group_ delimiter and preserve the full suffix as the solver/agent identifier. For records that do not contain that delimiter, retain the previous final-underscore-token fallback for compatibility with older/custom run-group IDs.

Regression coverage

Added a JSONL-level regression using the built-in direct_submission_solver naming shape and verified that parse_run_data() returns the full direct_submission_solver key and retains the corresponding paper run.

Risk

Low. Standard run groups now preserve exactly the shortname used when they were created. Legacy/custom IDs without the _run-group_ delimiter keep the previous parsing behavior.

Issue number

N/A. Found while auditing PaperBench metrics attribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant