Skip to content

Fix PaperBench rubric pruning at subtree boundaries - #160

Open
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-paperbench-prune-subtree-category
Open

Fix PaperBench rubric pruning at subtree boundaries#160
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-paperbench-prune-subtree-category

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

  • assign the existing Subtree task category when pruning an internal rubric node into a leaf
  • preserve the original task category when the pruned node is already a leaf
  • keep rubric IDs, requirements, weights, and fine-grained categories unchanged

TaskNode requires every leaf to have a task category and every internal node to have no task category. prune_to_depth() currently copies self.task_category when replacing a node with a leaf. For an internal node that value is necessarily None, so pruning at the requested depth raises ValueError instead of producing a pruned tree.

Subtree is already a supported category used for approximate subtree grading, so it is the natural category for a collapsed internal node.

Regression coverage prunes an internal root at depth zero and verifies it becomes a valid Subtree leaf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant