Skip to content

Commit d028d27

Browse files
yl231claude
yl231
and
claude
authored
Yl231 sync leaderboard 2026 05 (#40)
* Sync leaderboard with RouterArena README (Weave, Auto, R2 entries) - leaderboard.json: add Weave Router (74.61) and Auto Router (70.05) entries; fix NotDiamond Arena 63.0 -> 57.29 (per RouterArena PR #34). - category_scores.json: add per-domain breakdowns for r2-router, auto_router, and weave-router. Computed locally from RouterArena full-split predictions joined with dataset Domain/Category/Difficulty metadata; robustness from the per-router robustness predictions. - routers.json: add Weave Router and Auto Router metadata. Drop the orphan universal_router entry (had no corresponding leaderboard row). - routers.yaml: append R2-Router (was only in routers.json), Weave Router, and Auto Router. - flip_labels/: add per-query flip labels for r2-router, auto_router, weave-router, and vllm-sr. Computed as model-selection equality between full and robustness predictions, universal-name normalized. Sqwish Router (RouterArena PR #94) is excluded; will be added in a follow-up once that submission's CI re-run completes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add Sqwish Router entry; default null per-domain cells to 0 - leaderboard.json: add Sqwish Router (RouterArena PR #94) at rank 1. Metrics from the /evaluate CI run on 2026-05-20: Arena 75.27, Accuracy 76.40, Cost/1K $0.18, Robustness 100.00, Opt 14.92/17.17/100.00. - category_scores.json: add per-domain entry for sqwish-router. Also refresh entries for r2-router, auto_router, weave-router so that difficulty-bucket cells with no samples default to 0 (matching the existing TypeScript schema) instead of null. The TS build was rejecting the prior null cells against CompareSubcategory.metrics. - routers.json + routers.yaml: add Sqwish Router metadata (closed-source, affiliation Sqwish, 5-model pool from the PR config). - flip_labels/flip_labels_sqwish-router.json: new (all-stable, since Sqwish picked the same model on every robustness pair). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Apply PR #97 optimality fix to all workflow-evaluated routers RouterArena PR #97 (merged into main 2026-05-19) re-applied the PR #95 optimality fix and re-ran the /evaluate workflow for the four routers with for_optimality entries. The earlier sync used the pre-PR-#97 buggy values; this commit updates them to match origin/main README. - Weave Router: OptSel 1.38 -> 1.27, OptCost 12.27 -> 12.10, OptAcc 100.00 -> 89.56 - R2-Router: OptSel 32.38 -> 24.51, OptCost 74.16 -> 48.70, OptAcc 100.00 -> 99.85 - Auto Router: OptSel 43.77 -> 37.58, OptCost 42.29 -> 40.02, OptAcc 100.00 -> 86.04 - vLLM-SR: OptSel 94.10 -> 84.66, OptCost 90.12 -> 90.71, OptAcc 100.00 -> 89.24 Sqwish Router was evaluated by CI against PR #94's pre-#97 worktree, so its bot-reported numbers used the buggy formula. Recomputing locally with the merged main run.py on the same prediction file: - Sqwish Router: OptSel 14.92 -> 7.41, OptCost 17.17 -> 25.10, OptAcc 100.00 -> 90.47 Other rows (paper-era + Chayan) are unaffected since PR #97 only re-ran workflow-evaluated routers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Format data files with prettier (trailing newlines, inline single-element arrays) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: yl231 <yl231@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 21cdd26 commit d028d27

9 files changed

Lines changed: 12315 additions & 120 deletions

‎src/data/flip_labels/flip_labels_auto_router.json‎

Lines changed: 1682 additions & 0 deletions
Large diffs are not rendered by default.

‎src/data/flip_labels/flip_labels_r2-router.json‎

Lines changed: 1682 additions & 0 deletions
Large diffs are not rendered by default.

‎src/data/flip_labels/flip_labels_sqwish-router.json‎

Lines changed: 1682 additions & 0 deletions
Large diffs are not rendered by default.

‎src/data/flip_labels/flip_labels_vllm-sr.json‎

Lines changed: 1682 additions & 0 deletions
Large diffs are not rendered by default.

‎src/data/flip_labels/flip_labels_weave-router.json‎

Lines changed: 1682 additions & 0 deletions
Large diffs are not rendered by default.

‎src/data/routerMetrics/category_scores.json‎

Lines changed: 3728 additions & 0 deletions
Large diffs are not rendered by default.

‎src/data/routerMetrics/leaderboard.json‎

Lines changed: 113 additions & 91 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,69 @@
11
[
2+
{
3+
"Router Name": "Sqwish Router",
4+
"Arena Score": 75.27,
5+
"Optimal Selection Score": 7.41,
6+
"Optimal Cost Score": 25.1,
7+
"Optimal Acc. Score": 90.47,
8+
"Robustness Score": 100.0,
9+
"Latency Score": null,
10+
"Accuracy": 76.4,
11+
"Cost per 1k": 0.18
12+
},
213
{
314
"Router Name": "Weave Router",
415
"Arena Score": 74.61,
5-
"Optimal Selection Score": 1.38,
6-
"Optimal Cost Score": 12.27,
7-
"Optimal Acc. Score": 100.00,
16+
"Optimal Selection Score": 1.27,
17+
"Optimal Cost Score": 12.1,
18+
"Optimal Acc. Score": 89.56,
819
"Robustness Score": 79.05,
920
"Latency Score": null,
1021
"Accuracy": 78.43,
1122
"Cost per 1k": 0.92
1223
},
1324
{
1425
"Router Name": "R2-Router",
15-
"Arena Score": 71.60,
16-
"Optimal Selection Score": 32.38,
17-
"Optimal Cost Score": 74.16,
18-
"Optimal Acc. Score": 100.00,
26+
"Arena Score": 71.6,
27+
"Optimal Selection Score": 24.51,
28+
"Optimal Cost Score": 48.7,
29+
"Optimal Acc. Score": 99.85,
1930
"Robustness Score": 45.71,
2031
"Latency Score": null,
2132
"Accuracy": 71.23,
2233
"Cost per 1k": 0.06
2334
},
2435
{
25-
"Router Name": "RouterDC",
26-
"Arena Score": 33.75,
27-
"Optimal Selection Score": 39.84,
28-
"Optimal Cost Score": 72.998,
29-
"Optimal Acc. Score": 49.05,
30-
"Robustness Score": 85.24,
31-
"Latency Score": 10.7527,
32-
"Accuracy": 32.01,
33-
"Cost per 1k": 0.07
36+
"Router Name": "Auto Router",
37+
"Arena Score": 70.05,
38+
"Optimal Selection Score": 37.58,
39+
"Optimal Cost Score": 40.02,
40+
"Optimal Acc. Score": 86.04,
41+
"Robustness Score": 49.52,
42+
"Latency Score": null,
43+
"Accuracy": 70.17,
44+
"Cost per 1k": 0.12
45+
},
46+
{
47+
"Router Name": "vllm",
48+
"Arena Score": 67.23,
49+
"Optimal Selection Score": 84.66,
50+
"Optimal Cost Score": 90.71,
51+
"Optimal Acc. Score": 89.24,
52+
"Robustness Score": 90.95,
53+
"Latency Score": null,
54+
"Accuracy": 66.53,
55+
"Cost per 1k": 0.06
56+
},
57+
{
58+
"Router Name": "mirt_bert",
59+
"Arena Score": 66.89,
60+
"Optimal Selection Score": 3.44,
61+
"Optimal Cost Score": 19.6178,
62+
"Optimal Acc. Score": 78.18,
63+
"Robustness Score": 61.19,
64+
"Latency Score": 27.027,
65+
"Accuracy": 66.88,
66+
"Cost per 1k": 0.15
3467
},
3568
{
3669
"Router Name": "azure",
@@ -44,15 +77,15 @@
4477
"Cost per 1k": 0.54
4578
},
4679
{
47-
"Router Name": "carrot",
48-
"Arena Score": 63.87,
49-
"Optimal Selection Score": 2.68,
50-
"Optimal Cost Score": 6.7697,
51-
"Optimal Acc. Score": 78.63,
52-
"Robustness Score": 89.05,
53-
"Latency Score": 1.4993,
54-
"Accuracy": 67.21,
55-
"Cost per 1k": 2.06
80+
"Router Name": "nirt_bert",
81+
"Arena Score": 66.12,
82+
"Optimal Selection Score": 3.83,
83+
"Optimal Cost Score": 14.039,
84+
"Optimal Acc. Score": 77.88,
85+
"Robustness Score": 49.29,
86+
"Latency Score": 10.4167,
87+
"Accuracy": 66.34,
88+
"Cost per 1k": 0.21
5689
},
5790
{
5891
"Router Name": "gpt5",
@@ -66,41 +99,41 @@
6699
"Cost per 1k": 10.02
67100
},
68101
{
69-
"Router Name": "graphrouter",
70-
"Arena Score": 57.22,
71-
"Optimal Selection Score": 4.73,
72-
"Optimal Cost Score": 38.3347,
73-
"Optimal Acc. Score": 74.25,
74-
"Robustness Score": 94.29,
75-
"Latency Score": 2.6954,
76-
"Accuracy": 57.0,
77-
"Cost per 1k": 0.34
102+
"Router Name": "carrot",
103+
"Arena Score": 63.87,
104+
"Optimal Selection Score": 2.68,
105+
"Optimal Cost Score": 6.7697,
106+
"Optimal Acc. Score": 78.63,
107+
"Robustness Score": 89.05,
108+
"Latency Score": 1.4993,
109+
"Accuracy": 67.21,
110+
"Cost per 1k": 2.06
78111
},
79112
{
80-
"Router Name": "mirt_bert",
81-
"Arena Score": 66.89,
82-
"Optimal Selection Score": 3.44,
83-
"Optimal Cost Score": 19.6178,
84-
"Optimal Acc. Score": 78.18,
85-
"Robustness Score": 61.19,
86-
"Latency Score": 27.027,
87-
"Accuracy": 66.88,
88-
"Cost per 1k": 0.15
113+
"Router Name": "chayan",
114+
"Arena Score": 63.83,
115+
"Optimal Selection Score": 43.03,
116+
"Optimal Cost Score": 43.75,
117+
"Optimal Acc. Score": 88.74,
118+
"Robustness Score": null,
119+
"Latency Score": null,
120+
"Accuracy": 64.89,
121+
"Cost per 1k": 0.56
89122
},
90123
{
91-
"Router Name": "nirt_bert",
92-
"Arena Score": 66.12,
93-
"Optimal Selection Score": 3.83,
94-
"Optimal Cost Score": 14.039,
95-
"Optimal Acc. Score": 77.88,
96-
"Robustness Score": 49.29,
97-
"Latency Score": 10.4167,
98-
"Accuracy": 66.34,
99-
"Cost per 1k": 0.21
124+
"Router Name": "routerbench_mlp",
125+
"Arena Score": 57.56,
126+
"Optimal Selection Score": 13.39,
127+
"Optimal Cost Score": 24.4499,
128+
"Optimal Acc. Score": 83.32,
129+
"Robustness Score": 80.0,
130+
"Latency Score": 90.9091,
131+
"Accuracy": 61.62,
132+
"Cost per 1k": 4.83
100133
},
101134
{
102135
"Router Name": "notdiamond",
103-
"Arena Score": 63.0,
136+
"Arena Score": 57.29,
104137
"Optimal Selection Score": 1.55,
105138
"Optimal Cost Score": 2.1367,
106139
"Optimal Acc. Score": 76.81,
@@ -110,15 +143,15 @@
110143
"Cost per 1k": 4.1
111144
},
112145
{
113-
"Router Name": "routellm",
114-
"Arena Score": 48.07,
115-
"Optimal Selection Score": 99.72,
116-
"Optimal Cost Score": 99.6314,
117-
"Optimal Acc. Score": 68.76,
118-
"Robustness Score": 100.00,
119-
"Latency Score": 0.4016,
120-
"Accuracy": 47.04,
121-
"Cost per 1k": 0.27
146+
"Router Name": "graphrouter",
147+
"Arena Score": 57.22,
148+
"Optimal Selection Score": 4.73,
149+
"Optimal Cost Score": 38.3347,
150+
"Optimal Acc. Score": 74.25,
151+
"Robustness Score": 94.29,
152+
"Latency Score": 2.6954,
153+
"Accuracy": 57.0,
154+
"Cost per 1k": 0.34
122155
},
123156
{
124157
"Router Name": "routerbench_knn",
@@ -132,36 +165,25 @@
132165
"Cost per 1k": 4.27
133166
},
134167
{
135-
"Router Name": "routerbench_mlp",
136-
"Arena Score": 57.56,
137-
"Optimal Selection Score": 13.39,
138-
"Optimal Cost Score": 24.4499,
139-
"Optimal Acc. Score": 83.32,
140-
"Robustness Score": 80.00,
141-
"Latency Score": 90.9091,
142-
"Accuracy": 61.62,
143-
"Cost per 1k": 4.83
144-
},
145-
{
146-
"Router Name": "vllm",
147-
"Arena Score": 67.23,
148-
"Optimal Selection Score": 94.10,
149-
"Optimal Cost Score": 90.12,
150-
"Optimal Acc. Score": 100.00,
151-
"Robustness Score": 90.95,
152-
"Latency Score": null,
153-
"Accuracy": 66.53,
154-
"Cost per 1k": 0.06
168+
"Router Name": "routellm",
169+
"Arena Score": 48.07,
170+
"Optimal Selection Score": 99.72,
171+
"Optimal Cost Score": 99.6314,
172+
"Optimal Acc. Score": 68.76,
173+
"Robustness Score": 100.0,
174+
"Latency Score": 0.4016,
175+
"Accuracy": 47.04,
176+
"Cost per 1k": 0.27
155177
},
156178
{
157-
"Router Name": "chayan",
158-
"Arena Score": 63.83,
159-
"Optimal Selection Score": 43.03,
160-
"Optimal Cost Score": 43.75,
161-
"Optimal Acc. Score": 88.74,
162-
"Robustness Score": null,
163-
"Latency Score": null,
164-
"Accuracy": 64.89,
165-
"Cost per 1k": 0.56
179+
"Router Name": "RouterDC",
180+
"Arena Score": 33.75,
181+
"Optimal Selection Score": 39.84,
182+
"Optimal Cost Score": 72.998,
183+
"Optimal Acc. Score": 49.05,
184+
"Robustness Score": 85.24,
185+
"Latency Score": 10.7527,
186+
"Accuracy": 32.01,
187+
"Cost per 1k": 0.07
166188
}
167189
]

‎src/data/routers.json‎

Lines changed: 26 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,17 @@
11
{
2+
"Sqwish Router": {
3+
"name": "Sqwish Router",
4+
"type": "closed-source",
5+
"description": "Sqwish Router final submission.",
6+
"affiliation": "Sqwish",
7+
"modelPool": [
8+
"qwen/qwen3-235b-a22b-2507",
9+
"qwen/qwen3-next-80b-a3b-instruct",
10+
"Qwen/Qwen3-Coder-Next",
11+
"google/gemini-3.1-flash-lite",
12+
"deepseek/deepseek-v4-flash"
13+
]
14+
},
215
"Weave Router": {
316
"name": "Weave Router",
417
"type": "open-source",
@@ -128,27 +141,6 @@
128141
"websiteUrl": "https://arxiv.org/abs/2410.03834",
129142
"githubUrl": "https://github.com/ulab-uiuc/GraphRouter"
130143
},
131-
"universal_router": {
132-
"name": "Universal Router",
133-
"type": "open-source",
134-
"description": "Universal baseline that mirrors the RouterBench model mix",
135-
"affiliation": "RouterArena",
136-
"modelPool": [
137-
"WizardLM/WizardLM-13B-V1.2",
138-
"claude-instant-v1",
139-
"claude-v1",
140-
"claude-v2",
141-
"gpt-3.5-turbo-1106",
142-
"gpt-4-1106-preview",
143-
"meta/codellama-34b-instruct",
144-
"meta/llama-2-70b-chat",
145-
"mistralai/mistral-7b-chat",
146-
"mistralai/mixtral-8x7b-chat",
147-
"zero-one-ai/Yi-34B-Chat"
148-
],
149-
"paperUrl": "https://huggingface.co/datasets/withmartian/routerbench",
150-
"websiteUrl": "https://huggingface.co/datasets/withmartian/routerbench"
151-
},
152144
"mirt_bert": {
153145
"name": "MIRT-BERT",
154146
"type": "open-source",
@@ -226,26 +218,22 @@
226218
"openai/gpt-4.1-mini",
227219
"openai/gpt-4.1-nano",
228220
"openai/o1-mini",
229-
230221
"anthropic/claude-3-7-sonnet-20250219",
231222
"anthropic/claude-3-5-haiku-20241022",
232223
"anthropic/claude-3-haiku-20240307",
233224
"anthropic/claude-sonnet-4-20250514",
234-
235225
"google/gemini-1.5-pro-latest",
236226
"google/gemini-1.5-flash-latest",
237227
"google/gemini-2.0-flash-001",
238228
"google/gemini-2.5-flash",
239229
"google/gemini-2.5-pro",
240-
241230
"mistral/codestral-latest",
242231
"mistral/open-mixtral-8x7b",
243232
"mistral/mistral-large-latest",
244233
"mistral/mistral-medium-latest",
245234
"mistral/mistral-small-latest",
246235
"mistral/open-mistral-7b",
247236
"mistral/open-mistral-nemo",
248-
249237
"togetherai/Meta-Llama-3.1-8B-Instruct-Turbo",
250238
"togetherai/Meta-Llama-3.1-70B-Instruct-Turbo",
251239
"togetherai/Meta-Llama-3.1-405B-Instruct-Turbo"
@@ -258,10 +246,7 @@
258246
"type": "open-source",
259247
"description": "Binary selection between strong and weak models",
260248
"affiliation": "Berkeley",
261-
"modelPool": [
262-
"openai-gpt-4o",
263-
"mixtral-8x7b-instruct"
264-
],
249+
"modelPool": ["openai-gpt-4o", "mixtral-8x7b-instruct"],
265250
"paperUrl": "https://arxiv.org/abs/2406.18665",
266251
"websiteUrl": "https://arxiv.org/abs/2406.18665",
267252
"githubUrl": "https://github.com/lm-sys/RouteLLM",
@@ -347,5 +332,17 @@
347332
"paperUrl": "https://huggingface.co/adaptive-classifier/chayan",
348333
"websiteUrl": "https://huggingface.co/adaptive-classifier/chayan",
349334
"huggingfaceUrl": "https://huggingface.co/adaptive-classifier/chayan"
335+
},
336+
"Auto Router": {
337+
"name": "Auto Router",
338+
"type": "open-source",
339+
"description": "Cyclic baseline that rotates through its 4-model pool.",
340+
"affiliation": "RouterArena",
341+
"modelPool": [
342+
"deepseek/deepseek-v3.2",
343+
"qwen/qwen3-235b-a22b-2507",
344+
"qwen/qwen3.5-9b",
345+
"openai/gpt-4o"
346+
]
350347
}
351348
}

0 commit comments

Comments
 (0)