Summary
SWELancerEval.get_full_summary() initializes all three usage totals to 1 before adding the real token counts from each result:
summary["total_input_tokens"] = 1
summary["total_output_tokens"] = 1
summary["total_reasoning_tokens"] = 1
Every completed summary therefore over-reports each token category by exactly one.
Impact
Run-level usage accounting and any downstream cost/token analysis built from these summary fields is deterministically incorrect, including runs with no successfully parsed usage where the summary still reports one token in each category.
Proposed resolution
Initialize all three totals to 0 and keep the existing accumulation logic unchanged.
Add a focused summary regression with known grader-log token counts and verify the totals equal the exact sums with no offset.
Summary
SWELancerEval.get_full_summary()initializes all three usage totals to1before adding the real token counts from each result:Every completed summary therefore over-reports each token category by exactly one.
Impact
Run-level usage accounting and any downstream cost/token analysis built from these summary fields is deterministically incorrect, including runs with no successfully parsed usage where the summary still reports one token in each category.
Proposed resolution
Initialize all three totals to
0and keep the existing accumulation logic unchanged.Add a focused summary regression with known grader-log token counts and verify the totals equal the exact sums with no offset.