Skip to content

Fix PaperBench code-only fallback for non-code rubrics - #164

Open
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-paperbench-code-only-fallback
Open

Fix PaperBench code-only fallback for non-code rubrics#164
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-paperbench-code-only-fallback

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

  • make the code_only=True fallback produce a valid single Code Development task when a rubric contains no Code Development leaves
  • avoid constructing invalid intermediate TaskNode states
  • preserve the existing pruning path when Code Development criteria are present

run_judge() currently falls back with:

task_tree.set_task_category("Code Development").set_sub_tasks([])

when task_tree.code_only() returns None. TaskNode validation forbids a task category on an internal node, so the first call raises before set_sub_tasks([]) can run. Reversing the calls has the analogous problem because a leaf with no task category is also invalid.

The fix performs the fallback atomically with dataclasses.replace, creating the intended leaf without passing through an invalid intermediate tree.

Regression coverage verifies both the no-code fallback and the existing path where a Code Development subtree is present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant