Leaderboard scoring bias: runs with fewer tasks rank higher than comprehensive runs

[JoeProAI]

Summary

The leaderboard ranks agents by mean score across completed tasks, but does not account for the number of tasks attempted. This creates a systematic bias where agents running fewer (often easier, automated-only) tasks can outrank agents that complete the full 23-task suite.

Example

• Agent A runs 1 task, scores 100% → leaderboard rank: Add MIT license #1
• Agent B runs 9 tasks (all automated), scores 95.1% → leaderboard rank: Restructure as OpenClaw skill for ClawHub #2
• Agent C runs all 23 tasks (including judge-scored creative tasks like image generation, humanizing text, market research, email triage), scores 94.8% → leaderboard rank: Add friendly README with quick start and overview #3

Agent C demonstrably has broader capability, but ranks lowest. The creative/judge-scored tasks are inherently harder to score perfectly on, so agents that attempt them are penalized relative to those that skip them.

Suggested Fixes

1. Normalize by task count — weight the score by tasks_completed / total_tasks so comprehensive runs are rewarded
2. Require minimum task count for ranking — e.g., only rank runs with ≥N tasks (or suite=all)
3. Separate leaderboards — group by suite or task count range so comparisons are apples-to-apples
4. Display task count prominently — at minimum, show how many tasks each run completed so viewers can judge for themselves

Why This Matters

PinchBench is a great benchmark, but if the leaderboard incentivizes cherry-picking easy tasks, it undermines the signal. Agents should be rewarded for attempting the full suite, not penalized for it.

Thanks for building this — just want to help make the rankings more meaningful.

[JoeProAI]

This ties into the discussion in #47 about handling different providers and hardware on the leaderboard. The core point: normalizing by task count (or at minimum showing task count alongside score) would make the leaderboard more useful for everyone comparing across different setups.

[olearycrew]

Yes this is part of why I have the "official" concept so that I can control what we show by default.

Much like Moltbook (which this was in the style of) anyone can submit anything. So the biggest change here might be to when redesigning the layout include a filter for "full" runs

[JoeProAI]

This scoring bias is real: averaging only over completed tasks makes partial runs look artificially strong.

Proposal (pick one as the default leaderboard policy):

1. Full-run ranking: only rank submissions that ran the full suite (or a defined official suite). Show partial runs in a separate section.
2. Strict score: treat missing tasks as 0.0 and compute total_points / total_possible_points, so more tasks can only help if they succeed.

Implementation details that would help:

• Store tasks_total and tasks_completed in the submission.
• Display both: strict_score (missing=0) and partial_score (avg over completed), but sort by strict_score.
• Add a UI filter: hide partial runs (default on).

Right now the incentive is to run 2 easy tasks and look like a hero, while full 23-task runs get punished for being comprehensive.

[olearycrew]

Ah was that an AI answer. It smells like one.

I don't hate that but I think that's a v2 thing....and again is why we filter for official results