Skip to content

Propagate PaperBench judge validity through aggregate nodes - #145

Open
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/paperbench-valid-score-propagation
Open

Propagate PaperBench judge validity through aggregate nodes#145
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/paperbench-valid-score-propagation

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Make aggregate PaperBench judge nodes reflect grading failures in their descendants instead of always claiming a valid score.

Leaf/subtree grading exceptions are already converted to GradedTaskNode values with valid_score=False, but every internal node built after recursively grading children is currently hard-coded to valid_score=True. The final root can therefore appear valid while containing one or more judge failures.

Fixes #144.

Fix

Set aggregate validity from the children:

valid_score = all(child.valid_score for child in graded_sub_tasks)

The weighted score calculation is unchanged.

Regression coverage

Adds async tests verifying that:

  • one successful child plus one child whose grading raises produces the same weighted score but an invalid aggregate/root grade;
  • the successful child remains valid while the failed child is invalid;
  • an all-valid tree still produces valid_score=True.

No grading prompts, exception handling, score weights, or leaf-grade semantics are changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PaperBench aggregate nodes can hide invalid child grades

1 participant