Summary
Judge.grade() marks leaf/subtree failures with valid_score=False, but every internal node created after recursively grading children is hard-coded to valid_score=True.
A root grade can therefore report itself as valid even when one or more descendants failed to grade and were replaced with invalid zero-score nodes.
Current behavior
On a child grading exception:
return GradedTaskNode.from_task(
task,
score=0.0,
valid_score=False,
...
)After collecting children, the parent is constructed with:
regardless of graded_sub_tasks.
Impact
Judge failures can be hidden by aggregate nodes, including the final root result. Consumers that use valid_score to distinguish legitimate low scores from grading errors can treat an error-tainted PaperBench grade as valid.
Proposed resolution
An aggregate node should be valid only when all of its graded children are valid:
valid_score = all(child.valid_score for child in graded_sub_tasks)
Add a regression with one successful child and one child raising during grading, verifying the failing child and every aggregate ancestor are marked invalid while the weighted score behavior remains unchanged.
Summary
Judge.grade()marks leaf/subtree failures withvalid_score=False, but every internal node created after recursively grading children is hard-coded tovalid_score=True.A root grade can therefore report itself as valid even when one or more descendants failed to grade and were replaced with invalid zero-score nodes.
Current behavior
On a child grading exception:
After collecting children, the parent is constructed with:
regardless of
graded_sub_tasks.Impact
Judge failures can be hidden by aggregate nodes, including the final root result. Consumers that use
valid_scoreto distinguish legitimate low scores from grading errors can treat an error-tainted PaperBench grade as valid.Proposed resolution
An aggregate node should be valid only when all of its graded children are valid:
Add a regression with one successful child and one child raising during grading, verifying the failing child and every aggregate ancestor are marked invalid while the weighted score behavior remains unchanged.