Skip to content

Load legacy PaperBench reproduction metadata with optional defaults - #143

Open
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/paperbench-repro-metadata-compat
Open

Load legacy PaperBench reproduction metadata with optional defaults#143
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/paperbench-repro-metadata-compat

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Keep cached PaperBench reproduction metadata loadable when it was written before newer optional fields were added.

ReproductionMetadata already declares retried_results with an empty-list default and three fields as optional, but from_dict() indexes all four keys directly. Reusing an older *_executed_metadata.json can therefore raise ValueError even though the dataclass has well-defined defaults for the missing fields.

Fixes #142.

Fix

  • default missing retried_results to [];
  • load repro_execution_time, git_status_after_reproduce, and executed_submission with dict.get();
  • keep the core reproduction fields required as before.

Regression coverage

Adds tests verifying that:

  • a legacy dictionary containing only the core required fields loads successfully with the dataclass defaults;
  • newer optional fields and retried-run metadata are preserved when present.

Serialization output and the required core metadata schema are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cached PaperBench reproduction metadata cannot load older optional schemas

1 participant