Summary
ReproductionMetadata defines several fields with backwards-compatible defaults:
retried_results defaults to an empty list;repro_execution_time is optional;git_status_after_reproduce is optional;executed_submission is optional.
ReproductionMetadata.from_dict() nevertheless indexes all four keys directly. Metadata produced before one of these fields existed therefore raises ValueError("Missing required field in reproduction metadata") instead of using the dataclass defaults.
This path is exercised when PaperBench reuses an existing reproduced submission: _run_reproduce() reads the cached *_executed_metadata.json and calls ReproductionMetadata.from_dict().
Impact
Previously generated/cached reproduction outputs can become unusable after metadata schema additions, forcing unnecessary reproduction or causing grading to fail when overwrite_existing_output=False.
Proposed resolution
Keep the core reproduction fields required, but load defaulted/optional fields with dict.get() and default retried_results to [].
Add regressions for a legacy dictionary containing only the core required fields and for preservation of newer optional fields when present.
Summary
ReproductionMetadatadefines several fields with backwards-compatible defaults:retried_resultsdefaults to an empty list;repro_execution_timeis optional;git_status_after_reproduceis optional;executed_submissionis optional.ReproductionMetadata.from_dict()nevertheless indexes all four keys directly. Metadata produced before one of these fields existed therefore raisesValueError("Missing required field in reproduction metadata")instead of using the dataclass defaults.This path is exercised when PaperBench reuses an existing reproduced submission:
_run_reproduce()reads the cached*_executed_metadata.jsonand callsReproductionMetadata.from_dict().Impact
Previously generated/cached reproduction outputs can become unusable after metadata schema additions, forcing unnecessary reproduction or causing grading to fail when
overwrite_existing_output=False.Proposed resolution
Keep the core reproduction fields required, but load defaulted/optional fields with
dict.get()and defaultretried_resultsto[].Add regressions for a legacy dictionary containing only the core required fields and for preservation of newer optional fields when present.