Milestones
List view
Final launch gate: stable rules, benchmark artifacts, leaderboard confidence, and contributor-facing documentation for public miner participation.
No due dateRun the miner flow end-to-end on a small trusted cohort: reruns, queueing, leaderboard publication, and maintainer operations.
No due date•1/1 issues closedScoring integrity, benchmark parity, and protocol/docs alignment needed before miner competition results can be trusted.
No due date•5/5 issues closedAdd SWE-bench Verified support and its sandboxed patch evaluation runner.
No due date•2/2 issues closedRefactor the hidden benchmark builder and separate cached scoring from expensive execution.
No due date•2/2 issues closedAdd MMLU-Pro and explicit LiveCodeBench v6 support, then freeze the sampling protocol.
No due date•3/3 issues closedCore adapter registry, refactor existing benchmarks, and standardize benchmark item schema.
No due date•4/4 issues closed