Skip to content

Commit 80c81fb

Browse files
committed
package: report mean_score over the full benchmark (200), not attempted
Matches the leaderboard, where unattempted tasks count as 0; keeps the console summary from overstating a partial submission's mean.
1 parent 895320e commit 80c81fb

1 file changed

Lines changed: 3 additions & 3 deletions

File tree

‎src/programbench/submission.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -183,10 +183,10 @@ def aggregate(scores: dict[str, float], n_total: int) -> Headline:
183183
if not values:
184184
raise ValueError("No scored instances found")
185185
n = len(values)
186-
# mean is over attempted instances; resolved/near are over the full benchmark
187-
# (an unattempted task counts as unresolved).
186+
# mean, resolved, and near are all over the full benchmark — an unattempted task counts
187+
# as 0, matching how the leaderboard scores partial submissions.
188188
return Headline(
189-
mean_score=round(sum(values) / n, 4),
189+
mean_score=round(sum(values) / n_total, 4),
190190
resolved_pct=round(100 * sum(s >= RESOLVED_THRESHOLD for s in values) / n_total, 1),
191191
near_resolved_pct=round(100 * sum(s >= NEAR_RESOLVED_THRESHOLD for s in values) / n_total, 1),
192192
n_instances_attempted=n,

0 commit comments

Comments
 (0)