Release 0.7.20 routing selection cleanup

Author: anjieyang

README.md

@@ -4,11 +4,13 @@
 
 <h1>UncommonRoute</h1>
 
-**Cut your LLM costs by 82% with automatic model routing.**
+**Automatic model routing for lower LLM spend.**
 
 Most of your LLM budget goes to simple tasks that don't need a premium model.
 UncommonRoute picks the cheapest model that still gets the job done — automatically.
 
+Current held-out eval: **91.8% task pass rate** with an **81.9 cost-savings score** on CommonRouterBench.
+
 <br>
 
 <a href="https://pypi.org/project/uncommon-route/"><img src="https://img.shields.io/pypi/v/uncommon-route?style=flat-square&logo=pypi&logoColor=white&label=PyPI" alt="PyPI"></a>
@@ -42,6 +44,8 @@ pipx install uncommon-route
 
 `pipx` is the best default for most CLI users: it installs UncommonRoute into its own isolated environment, keeps your system Python clean, and gives you a clean uninstall path.
 
+A normal install includes the trained v2 runtime assets and embedding dependencies. You do not need a separate `[v2]` install for production routing.
+
 If you do not have `pipx` yet, prefer your OS package manager when it is available (`brew install pipx` on macOS, `sudo apt install pipx` on recent Ubuntu, `sudo dnf install pipx` on Fedora), then run `pipx ensurepath`.
 
 If that is not available, see the [pipx installation guide](https://pipx.pypa.io/stable/installation/) or install it with:
@@ -138,26 +142,30 @@ Auto-routing will only consider models backed by a registered provider.
 
 ## How It Works
 
-Every request is analyzed by three independent signals, then routed to the cheapest capable model:
+Every request is analyzed by multiple local signals, then routed to the cheapest capable model available from your configured upstream:
 
 ```
-"hello"                              → 🟢 nano         $0.0008
-"fix the typo on line 3"             → 🟢 deepseek     $0.0012
-"refactor this 500-line module"      → 🟠 sonnet       $0.0337
-"design a distributed scheduler"     → 🔴 opus         $0.0562
+"hello"                              → economy tier
+"fix the typo on line 3"             → economy / balanced tier
+"refactor this 500-line module"      → balanced / premium tier
+"design a distributed scheduler"     → premium tier
 ```
 
+Actual model IDs and prices come from the live upstream model catalog plus your local overrides. UncommonRoute does not rely on a single hardcoded model list.
+
 | Signal | What it does | Speed (CPU, warm) |
 |---|---|---|
 | **Metadata** | Conversation structure, tool usage, depth | <1ms |
-| **Embedding** | Semantic similarity to known task patterns (bge-small) | ~20ms |
-| **Structural** | Text complexity features (shadow mode) | <1ms |
+| **Embedding** | Trained BGE classifier over the user request, recent agent state, and metadata; KNN fallback when uncertain | ~25–35ms end-to-end warm route overhead |
+| **Structural** | Text and conversation complexity; active on selected requests, shadow-tracked otherwise | <1ms |
+
+End-to-end `route()` overhead on a warm process is typically **~25–35ms** on CPU and is dominated by the embedding signal. Cold start includes loading the embedding model and can take seconds on a fresh process or machine; after warmup, routing stays local.
 
-End-to-end `route()` overhead on a warm process is **~20–25ms** (dominated by the embedding signal). Cold start is a few hundred ms for the first request. GPU or a cached embedding path can bring this under 5ms.
+Signals vote. The ensemble picks the tier. The router then selects the cheapest model that satisfies tier, capability, transport, and upstream availability constraints. Unknown or dynamic upstream pricing is treated conservatively instead of being interpreted as a real negative price.
 
-Signals vote. The ensemble picks the tier. The router selects the cheapest model in that tier. If uncertain, it leans conservative — better to spend a little more than to fail the task.
+Routing is **per request / per agent step**, not sticky for an entire session. Protocol-level constraints still apply when the request requires them, for example Anthropic thinking continuations.
 
-**It gets smarter over time.** Signal weights adjust from routing outcomes. The embedding index grows with usage. Low-confidence predictions automatically escalate.
+**It gets smarter over time.** Local feedback can adjust signal weights, high-confidence agreement can grow the embedding index, and low-confidence predictions escalate instead of silently under-routing.
 
 ---
 
@@ -171,27 +179,31 @@ We didn't patch it. We rebuilt from scratch.
 
 | | v1 | v2 |
 |---|---|---|
-| **Accuracy** | 43% | **78%** |
-| **Task pass rate** | 100% (cheated — always chose most expensive) | **93.4%** (real routing) |
-| **Cost savings** | 0% | **82%** |
+| **Tier match accuracy** | 43% | **74.0%** held-out |
+| **Task pass rate** | 100% (cheated — always chose most expensive) | **91.8%** with real routing |
+| **Cost-savings score** | 0% | **81.9** |
 
 We're telling you this because we'd rather you trust our numbers than be impressed by them.
 
 ---
 
 ## Benchmarks
 
-Tested on [CommonRouterBench](https://github.com/CommonstackAI/CommonRouterBench) — 970 real agent task traces across SWE-Bench, BFCL, MT-RAG, QMSum, and PinchBench. All numbers measured end-to-end through the production code path.
+Tested on [CommonRouterBench](https://github.com/CommonstackAI/CommonRouterBench) — 970 real agent task traces across SWE-Bench, BFCL, MT-RAG, QMSum, and PinchBench. The public numbers below use the 196-row held-out split, not the training or calibration rows.
 
 | Metric | Value |
 |---|---|
-| **Cost savings** | **82%** vs always-premium |
-| **Task pass rate** | **93.4%** |
-| **Routing overhead** | **~20–25ms** (warm process, CPU, bge-small embedding) |
-| **Accuracy** | **78%** tier match |
+| **Task pass rate** | **91.8%** |
+| **Tier match accuracy** | **74.0%** |
+| **Cost-savings score** | **81.9** vs always-premium baseline |
+| **Overall score** | **76.7** |
+| **Warm routing overhead** | **p50 25.6ms / p90 32.1ms** on a local CPU run |
 
 ```bash
-python scripts/eval_v2.py  # reproduce it yourself
+python -m pip install -e ".[dev]"
+python -m pip install "git+https://github.com/CommonstackAI/CommonRouterBench.git"
+python scripts/eval_v2.py --split holdout
+python scripts/bench_overhead.py --iterations 50 --json
 ```
 
 ---

README.zh-CN.md

@@ -4,11 +4,13 @@
 
 <h1>UncommonRoute</h1>
 
-**自动模型路由，节省 82% 的 LLM 开销。**
+**自动模型路由，降低 LLM 开销。**
 
 大部分 LLM 预算都花在了不需要顶配模型的简单任务上。
 UncommonRoute 自动选最便宜、又能完成任务的模型。
 
+当前 held-out 评测：CommonRouterBench 上 **91.8% 任务完成率**，**81.9 成本节省分数**。
+
 <br>
 
 <a href="https://pypi.org/project/uncommon-route/"><img src="https://img.shields.io/pypi/v/uncommon-route?style=flat-square&logo=pypi&logoColor=white&label=PyPI" alt="PyPI"></a>
@@ -42,6 +44,8 @@ pipx install uncommon-route
 
 对大多数 CLI 用户来说，`pipx` 是更好的默认方案：它会把 UncommonRoute 安装到独立环境里，不污染系统 Python，卸载也更干净。
 
+普通安装已经包含训练好的 v2 runtime assets 和 embedding 依赖。生产路由不需要再额外安装 `[v2]`。
+
 如果你还没有安装 `pipx`，优先使用系统包管理器安装会更稳妥，比如 macOS 上用 `brew install pipx`，较新的 Ubuntu 用 `sudo apt install pipx`，Fedora 用 `sudo dnf install pipx`，然后执行 `pipx ensurepath`。
 
 如果系统里没有现成的 `pipx` 包，再参考 [pipx 官方安装文档](https://pipx.pypa.io/stable/installation/) 或直接运行：
@@ -138,24 +142,30 @@ Auto 路由只会在已注册的 provider 范围内选模型。
 
 ## 工作原理
 
-每个请求经过三个独立信号的分析，然后路由到最便宜的合适模型：
+每个请求会先经过多个本地信号分析，然后路由到你配置的 upstream 里最便宜的合适模型：
 
 ```
-"你好"                     → 🟢 nano         $0.0008
-"修一下第 3 行的拼写错误"   → 🟢 deepseek     $0.0012
-"重构这个 500 行的模块"    → 🟠 sonnet       $0.0337
-"设计一个分布式调度系统"    → 🔴 opus         $0.0562
+"你好"                     → economy tier
+"修一下第 3 行的拼写错误"   → economy / balanced tier
+"重构这个 500 行的模块"    → balanced / premium tier
+"设计一个分布式调度系统"    → premium tier
 ```
 
+具体模型 ID 和价格来自当前 upstream 的模型目录以及你的本地配置。UncommonRoute 不依赖单一写死的模型列表。
+
 | 信号 | 分析内容 | 耗时 |
 |---|---|---|
 | **元数据信号** | 对话轮次、工具调用、上下文深度 | <1ms |
-| **语义信号** | 与已知任务模式的相似度（embedding） | ~20ms |
-| **结构信号** | 文本复杂度特征（影子模式运行） | <1ms |
+| **语义信号** | 基于 BGE 的训练分类器，结合用户请求、最近 agent 状态和元数据；不确定时回退到 KNN | warm route 端到端约 ~25–35ms |
+| **结构信号** | 文本和对话复杂度；部分请求直接参与，其他请求以 shadow 方式追踪 | <1ms |
+
+warm process 上，端到端 `route()` 开销通常是 **~25–35ms**（CPU），主要来自语义信号。冷启动需要加载 embedding 模型，在全新进程或新机器上可能需要数秒；预热后路由都在本地完成。
+
+多个信号投票，集成模型决定 tier。随后路由器会在满足 tier、能力、协议和 upstream 可用性约束的模型中选择最低成本的模型。未知价格或动态价格会被保守处理，不会被当成真实负价格。
 
-三个信号投票，集成模型决定 tier，路由器在对应 tier 中选最便宜的模型。遇到不确定的情况，宁可多花一点也不冒失败风险。
+路由是 **per request / per agent step** 的，不会把整个 session 锁死在同一个模型上。只有协议本身要求连续性的场景才会加约束，比如 Anthropic thinking continuation。
 
-**越用越准。** 信号权重根据实际路由效果自动调整，向量索引随使用量增长，低置信度预测自动升级。
+**越用越准。** 本地反馈可以调整信号权重，高置信度一致判断可以扩展 embedding index，低置信度预测会升级，而不是静默降到不合适的模型。
 
 ---
 
@@ -169,27 +179,31 @@ v1 分类器在干净的 benchmark 上跑出了 88.5% 的准确率，我们直
 
 | | v1 | v2 |
 |---|---|---|
-| **准确率** | 43% | **78%** |
-| **任务完成率** | 100%（作弊——永远选最贵的） | **93.4%**（真正的路由决策） |
-| **成本节省** | 0% | **82%** |
+| **Tier match 准确率** | 43% | **74.0%** held-out |
+| **任务完成率** | 100%（作弊——永远选最贵的） | **91.8%**（真正的路由决策） |
+| **成本节省分数** | 0% | **81.9** |
 
 我们把这些数据告诉你，是因为比起让你被数字震撼到，我们更希望你能信任它。
 
 ---
 
 ## 性能数据
 
-基于 [CommonRouterBench](https://github.com/CommonstackAI/CommonRouterBench) 测试——970 条真实 Agent 任务轨迹，覆盖 SWE-Bench、BFCL、MT-RAG、QMSum 和 PinchBench。所有数据通过生产代码端到端测量。
+基于 [CommonRouterBench](https://github.com/CommonstackAI/CommonRouterBench) 测试——970 条真实 Agent 任务轨迹，覆盖 SWE-Bench、BFCL、MT-RAG、QMSum 和 PinchBench。下面的公开数字使用 196 条 held-out split，不包含训练集和 calibration 集。
 
 | 指标 | 数值 |
 |---|---|
-| **成本节省** | **82%**（对比全程顶配） |
-| **任务完成率** | **93.4%** |
-| **路由延迟** | **~20–25ms**（warm process, CPU, bge-small embedding） |
-| **准确率** | **78%** tier 匹配 |
+| **任务完成率** | **91.8%** |
+| **Tier match 准确率** | **74.0%** |
+| **成本节省分数** | **81.9**（对比全程顶配 baseline） |
+| **综合分数** | **76.7** |
+| **warm 路由延迟** | 本地 CPU 运行 **p50 25.6ms / p90 32.1ms** |
 
 ```bash
-python scripts/eval_v2.py  # 自己跑一遍看看
+python -m pip install -e ".[dev]"
+python -m pip install "git+https://github.com/CommonstackAI/CommonRouterBench.git"
+python scripts/eval_v2.py --split holdout
+python scripts/bench_overhead.py --iterations 50 --json
 ```
 
 ---

openclaw-plugin/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@anjieyang/uncommon-route",
-  "version": "0.7.19",
+  "version": "0.7.20",
   "description": "OpenClaw plugin for UncommonRoute, the local LLM router that cuts premium-model spend",
   "type": "module",
   "main": "src/index.js",

openclaw-plugin/src/index.js

@@ -16,7 +16,7 @@
 import { spawn, execSync } from "node:child_process";
 import { setTimeout as sleep } from "node:timers/promises";
 
-const VERSION = "0.7.14";
+const VERSION = "0.7.20";
 const DEFAULT_PORT = 8403;
 const DEFAULT_UPSTREAM = "";
 const HEALTH_TIMEOUT_MS = 15_000;

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "uncommon-route"
-version = "0.7.19"
+version = "0.7.20"
 description = "Local LLM router that cuts premium-model spend with adaptive 3-tier routing, OpenAI + Anthropic compatible"
 requires-python = ">=3.11"
 license = "MIT"

tests/test_quality_routing.py

@@ -1507,6 +1507,86 @@ def test_normal_tool_selection_does_not_force_medium_floor() -> None:
     assert features.tier_cap is None
 
 
+def test_suggestion_mode_prompt_is_simple_capped_side_channel() -> None:
+    from uncommon_route.proxy import _classify_step, _extract_routing_features
+
+    suggestion_prompt = (
+        "[SUGGESTION MODE: Suggest what the user might naturally type next into Claude Code.]\n\n"
+        "FIRST: Look at the user's recent messages and original request.\n"
+        "Your job is to predict what THEY would type - not what you think they should do.\n"
+        "Reply with ONLY the suggestion, no quotes or explanation."
+    )
+    body = {
+        "messages": [
+            {"role": "user", "content": "Build a small weather CLI."},
+            {"role": "assistant", "content": "I created the project files."},
+            {"role": "user", "content": suggestion_prompt},
+        ],
+        "tools": [{"type": "function", "function": {"name": "bash"}}],
+    }
+
+    step_type, tool_names = _classify_step(body)
+    features = _extract_routing_features(
+        body,
+        step_type=step_type,
+        tool_names=tool_names,
+        prompt=suggestion_prompt,
+    )
+
+    assert features.step_type == "tool-selection"
+    assert features.step_risk == "low"
+    assert features.tier_floor is None
+    assert features.tier_cap is Tier.SIMPLE
+    assert features.tier_cap_reason == "suggestion-mode"
+
+
+def test_short_tool_selection_task_is_not_high_risk_from_tool_presence_alone() -> None:
+    from uncommon_route.proxy import _classify_step, _extract_routing_features
+
+    prompt = "帮我创建一个新的 Python 项目目录，叫 weather-cli"
+    body = {
+        "messages": [{"role": "user", "content": prompt}],
+        "tools": [{"type": "function", "function": {"name": "bash"}} for _ in range(27)],
+    }
+
+    step_type, tool_names = _classify_step(body)
+    features = _extract_routing_features(
+        body,
+        step_type=step_type,
+        tool_names=tool_names,
+        prompt=prompt,
+    )
+
+    assert features.step_type == "tool-selection"
+    assert features.step_risk == "normal"
+    assert features.tier_floor is None
+
+
+def test_dense_tool_selection_task_can_still_be_high_risk_by_shape() -> None:
+    from uncommon_route.proxy import _classify_step, _extract_routing_features
+
+    prompt = (
+        "清空目录中的所有内容，我想做一个不需要 api 的命令行天气查询工具，输入城市名就能显示当前天气、"
+        "温度、湿度。帮我规划一下：用什么天气 API、项目结构怎么组织"
+    )
+    body = {
+        "messages": [{"role": "user", "content": prompt}],
+        "tools": [{"type": "function", "function": {"name": "bash"}} for _ in range(27)],
+    }
+
+    step_type, tool_names = _classify_step(body)
+    features = _extract_routing_features(
+        body,
+        step_type=step_type,
+        tool_names=tool_names,
+        prompt=prompt,
+    )
+
+    assert features.step_type == "tool-selection"
+    assert features.step_risk == "high"
+    assert features.tier_floor is Tier.MEDIUM
+
+
 def test_system_json_directive_sets_structured_output_floor() -> None:
     from uncommon_route.proxy import _classify_step, _extract_routing_features