Skip to content

Skip null PaperBench token usage during aggregation - #147

Open
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/paperbench-null-token-usage
Open

Skip null PaperBench token usage during aggregation#147
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/paperbench-null-token-usage

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Keep successful SimpleJudge runs from failing during post-grade token aggregation when a leaf has no OpenAI token-usage metadata.

SimpleJudge.grade_leaf() includes a token_usage key in judge_metadata, but generic/non-OpenAI completers can leave its value as None. The aggregation helper currently checks only whether the key exists and then passes the value to TokenUsage.from_dict(), which calls .items() and crashes on None.

Fixes #146.

Fix

Only deserialize leaf token usage when the metadata value is a dictionary. Null, missing, or otherwise unavailable usage contributes no token counts for that leaf.

Regression coverage

Adds tests verifying that:

  • a leaf with {"token_usage": None} is skipped without error;
  • normal usage from another leaf is still summed exactly;
  • a leaf with no token-usage key produces an empty total rather than failing.

TokenUsage serialization, OpenAI usage field mapping, and valid usage summation are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PaperBench token aggregation crashes on explicit null usage metadata

1 participant