Repository navigation
Commit 9d798ff
Add model price profiles; exact price matching; charge unmatched models the max price
- model_cost/model_profiles.yaml is now the source of truth for prices: one
profile per model with aliases; model_cost.json is generated from it by
scripts/pricing/build_model_cost.py (--check verifies it is in sync).
- Duplicate spellings are folded into aliases where that moves no score.
Three models stay double-profiled with a conflict note (qwen3-235b,
claude-haiku-4.5, qwen3-30b) because different board entries are charged
different prices for them today.
- Evaluator: price lookup is exact id/alias only (the substring fallback is
removed). Spellings that relied on it are added as explicit aliases.
- A model with no price profile is charged the highest input/output price in
the table instead of $0, and an unknown model name is graded and charged
rather than dropped. The submission check warns instead of failing.
- Re-pricing all 265,568 rows in router_inference/predictions gives identical
costs before and after.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>1 parent f371ef0 commit 9d798ff
7 files changed
Lines changed: 777 additions & 92 deletions
File tree
- llm_evaluation
- model_cost
- router_inference
- scripts/pricing
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| |||
87 | 88 | | |
88 | 89 | | |
89 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
90 | 95 | | |
91 | 96 | | |
92 | 97 | | |
| |||
207 | 212 | | |
208 | 213 | | |
209 | 214 | | |
210 | | - | |
211 | | - | |
212 | | - | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
213 | 221 | | |
214 | 222 | | |
215 | 223 | | |
216 | 224 | | |
217 | 225 | | |
218 | 226 | | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
225 | 248 | | |
226 | 249 | | |
227 | 250 | | |
| |||
248 | 271 | | |
249 | 272 | | |
250 | 273 | | |
251 | | - | |
252 | | - | |
253 | | - | |
254 | | - | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
255 | 283 | | |
256 | | - | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
257 | 289 | | |
258 | | - | |
259 | 290 | | |
260 | 291 | | |
261 | 292 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
362 | 362 | | |
363 | 363 | | |
364 | 364 | | |
365 | | - | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
366 | 368 | | |
367 | 369 | | |
368 | 370 | | |
369 | | - | |
370 | | - | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
371 | 375 | | |
372 | | - | |
| 376 | + | |
373 | 377 | | |
374 | 378 | | |
375 | 379 | | |
| |||
586 | 590 | | |
587 | 591 | | |
588 | 592 | | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
589 | 605 | | |
590 | 606 | | |
591 | 607 | | |
| |||
0 commit comments