Skip to content

Commit d9bce85

Browse files
yl231claude
andauthored
Re-add Weave Router (v0.54) and refresh Azure-Model-Router metrics (#44)
Syncs with RouterArena main after the no-training-policy retrain and the PR #98/#107 Azure re-evaluation. - Weave Router: re-added at rank 2 (RouterArena PR #109). v0.54 "clean retrain" — 6-model cluster routing trained only on OpenRouterBench labels + Artificial Analysis quality priors, no RouterArena prompts. Arena 72.82, Acc 76.32, Cost $0.94, Robustness 100.00; optimality metrics are blank (submission has no for_optimality entries). - Azure-Model-Router: refreshed to Arena 71.87, Acc 72.82, Cost $0.22, Robustness 71.43; optimality cleared to match README. Weave per-domain category_scores fall back to synthetic (no v0.54 prediction file imported); the stale flip_labels_weave-router.json is left in place. Metadata (model pool, description, links) updated to v0.54. Co-authored-by: yl231 <yl231@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent b631741 commit d9bce85

3 files changed

Lines changed: 48 additions & 27 deletions

File tree

‎src/data/routerMetrics/leaderboard.json‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,17 @@
1010
"Accuracy": 76.4,
1111
"Cost per 1k": 0.18
1212
},
13+
{
14+
"Router Name": "Weave Router",
15+
"Arena Score": 72.82,
16+
"Optimal Selection Score": null,
17+
"Optimal Cost Score": null,
18+
"Optimal Acc. Score": null,
19+
"Robustness Score": 100.0,
20+
"Latency Score": null,
21+
"Accuracy": 76.32,
22+
"Cost per 1k": 0.94
23+
},
1324
{
1425
"Router Name": "R2-Router",
1526
"Arena Score": 71.6,
@@ -56,14 +67,14 @@
5667
},
5768
{
5869
"Router Name": "azure",
59-
"Arena Score": 66.66,
60-
"Optimal Selection Score": 22.52,
61-
"Optimal Cost Score": 46.322,
62-
"Optimal Acc. Score": 81.96,
63-
"Robustness Score": 54.07,
70+
"Arena Score": 71.87,
71+
"Optimal Selection Score": null,
72+
"Optimal Cost Score": null,
73+
"Optimal Acc. Score": null,
74+
"Robustness Score": 71.43,
6475
"Latency Score": null,
65-
"Accuracy": 68.09,
66-
"Cost per 1k": 0.54
76+
"Accuracy": 72.82,
77+
"Cost per 1k": 0.22
6778
},
6879
{
6980
"Router Name": "nirt_bert",

‎src/data/routers.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,22 @@
1212
"deepseek/deepseek-v4-flash"
1313
]
1414
},
15+
"Weave Router": {
16+
"name": "Weave Router",
17+
"type": "open-source",
18+
"description": "Weave Router (v0.54): 6-model cluster routing trained only on OpenRouterBench labels + Artificial Analysis quality priors. No RouterArena prompts in any training signal.",
19+
"affiliation": "Weave",
20+
"modelPool": [
21+
"deepseek/deepseek-v4-flash",
22+
"deepseek/deepseek-v4-pro",
23+
"gemini-3.1-flash-lite-preview",
24+
"gemini-3.1-pro-preview",
25+
"Qwen/Qwen3-Coder-Next",
26+
"qwen/qwen3-next-80b-a3b-instruct"
27+
],
28+
"websiteUrl": "https://workweave.ai",
29+
"githubUrl": "https://github.com/workweave/router"
30+
},
1531
"R2-Router": {
1632
"name": "R2-Router",
1733
"type": "open-source",

‎src/data/routers.yaml‎

Lines changed: 14 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,17 @@
1-
# --- Weave Router temporarily removed from the public leaderboard. ---
2-
# Restore by uncommenting this block AND re-adding the matching entries
3-
# in leaderboard.json, routers.json, and category_scores.json (key:
4-
# "weave-router"). flip_labels/flip_labels_weave-router.json is kept.
5-
#
6-
# Weave Router:
7-
# name: Weave Router
8-
# type: open-source
9-
# description: Cluster-routing (v0.62) over a 7-model BYOK pool spanning Anthropic, OpenAI, Google, and OpenRouter providers. Embeds each prompt with Jina v2 INT8 ONNX (768-dim), runs top-p=4 cluster sum against per-cluster rankings trained on the RouterArena full split with k=160 clusters, then selects the cost-quality optimum via an alpha-blended score (alpha=0.25).
10-
# affiliation: Weave
11-
# modelPool:
12-
# - claude-opus-4-7
13-
# - gpt-5.5
14-
# - gemini-3.1-pro-preview
15-
# - gemini-3.1-flash-lite-preview
16-
# - deepseek/deepseek-v4-pro
17-
# - deepseek/deepseek-v4-flash
18-
# - qwen/qwen3-235b-a22b-2507
19-
# websiteUrl: https://weaverouter.com
20-
# githubUrl: https://github.com/workweave/router
1+
Weave Router:
2+
name: Weave Router
3+
type: open-source
4+
description: 'Weave Router (v0.54): 6-model cluster routing trained only on OpenRouterBench labels + Artificial Analysis quality priors. No RouterArena prompts in any training signal.'
5+
affiliation: Weave
6+
modelPool:
7+
- deepseek/deepseek-v4-flash
8+
- deepseek/deepseek-v4-pro
9+
- gemini-3.1-flash-lite-preview
10+
- gemini-3.1-pro-preview
11+
- Qwen/Qwen3-Coder-Next
12+
- qwen/qwen3-next-80b-a3b-instruct
13+
websiteUrl: https://workweave.ai
14+
githubUrl: https://github.com/workweave/router
2115

2216
RouterDC:
2317
name: RouterDC

0 commit comments

Comments
 (0)