Skip to content

Commit d1e5325

Browse files
leestottleestott
committed
update README and add live demo config + Azure Retail Pricing API
1 parent ecf0aa2 commit d1e5325

22 files changed

Lines changed: 817 additions & 48 deletions

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ htmlcov/
5555
coverage.xml
5656
*.cover
5757
.hypothesis/
58+
.cache/
5859

5960
# Logs
6061
*.log

‎README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,9 @@ cp .env.example .env
145145

146146
Edit `configs/default.yaml` to set endpoints, baseline model, pricing, and judge settings.
147147

148+
- **Azure retail pricing refresh** — supported Foundry model token prices are
149+
refreshed from the public Azure Retail Prices API and cached for 24 hours;
150+
YAML prices remain fallbacks
148151
- **24 models pre-configured** — Azure OpenAI, Anthropic, xAI, DeepSeek, Meta
149152
- **LLM-as-a-judge** enabled by default; configure via `AZURE_JUDGE_*` env vars or `judge.endpoint` in YAML
150153
- **Environment variables** (`${VAR}`) are resolved from `.env` at load time

‎configs/README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,10 @@
77
| [large_scale.yaml](large_scale.yaml) | All | Enabled | 1000+ prompts (higher concurrency, longer timeouts) |
88
| [foundry.yaml](foundry.yaml) | — | — | Foundry cloud eval settings (graders, thresholds) |
99

10-
Edit `default.yaml` to set your endpoint URLs, baseline model, and pricing. Environment variables (`${VAR}`) are resolved from `.env`.
10+
Edit `default.yaml` to set your endpoint URLs, baseline model, and pricing.
11+
Environment variables (`${VAR}`) are resolved from `.env`. Supported Foundry
12+
model prices are refreshed from the Azure Retail Prices API and cached under
13+
`.cache/`; YAML prices are retained as offline and ambiguity fallbacks.
1114

1215
## Prompt Templates
1316

‎configs/default.yaml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ endpoints:
2323
api_key: "${AZURE_OPENAI_KEY}"
2424
deployment_name: "${AZURE_BASELINE_DEPLOYMENT}"
2525
parameters:
26+
temperature: 0.7
2627
max_tokens: 1024
2728

2829
pricing: # USD per 1M tokens
@@ -121,6 +122,42 @@ pricing: # USD per 1M tokens
121122
input: 0.20
122123
output: 0.20
123124

125+
# Refresh supported Foundry model token prices from the public Azure Retail
126+
# Prices API. YAML values above remain explicit fallbacks for unavailable or
127+
# ambiguous meters and for Marketplace models.
128+
pricing_source:
129+
azure_retail:
130+
enabled: true
131+
currency: "USD"
132+
# Set this to the Azure region used by your Foundry resource when regional
133+
# retail records differ, for example "eastus".
134+
region: null
135+
cache_file: ".cache/azure-retail-prices.json"
136+
cache_ttl_hours: 24
137+
timeout_seconds: 20
138+
models:
139+
gpt-4.1:
140+
search: "GPT 4.1"
141+
excluded_terms: ["mini", "nano"]
142+
gpt-4.1-mini: "GPT 4.1 Mini"
143+
gpt-4.1-nano: "GPT 4.1 Nano"
144+
gpt-4o:
145+
search: "GPT-4o"
146+
excluded_terms: ["mini"]
147+
gpt-4o-mini: "GPT-4o mini"
148+
o4-mini: "o4-mini"
149+
gpt-5:
150+
search: "GPT 5"
151+
excluded_terms:
152+
[".1", ".2", ".4", ".5", ".6", "mini", "nano", "chat", "codex", "pro"]
153+
gpt-5-mini: "GPT 5 Mini"
154+
gpt-5-nano: "GPT 5 Nano"
155+
gpt-5-chat: "GPT 5 chat"
156+
gpt-5.2:
157+
search: "GPT 5.2"
158+
excluded_terms: ["pro", "chat", "ShortCo", "LongCo"]
159+
gpt-5.2-chat: "GPT 5.2 chat"
160+
124161
concurrency:
125162
max_parallel_requests: 5
126163
request_timeout_seconds: 60

‎configs/large_scale.yaml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,37 @@ pricing: # USD per 1M tokens
6363
# ── Meta ──
6464
Llama-4-Maverick-17B-128E-Instruct-FP8: { input: 0.20, output: 0.20 }
6565

66+
pricing_source:
67+
azure_retail:
68+
enabled: true
69+
currency: "USD"
70+
region: null
71+
cache_file: ".cache/azure-retail-prices.json"
72+
cache_ttl_hours: 24
73+
timeout_seconds: 20
74+
models:
75+
gpt-4.1:
76+
search: "GPT 4.1"
77+
excluded_terms: ["mini", "nano"]
78+
gpt-4.1-mini: "GPT 4.1 Mini"
79+
gpt-4.1-nano: "GPT 4.1 Nano"
80+
gpt-4o:
81+
search: "GPT-4o"
82+
excluded_terms: ["mini"]
83+
gpt-4o-mini: "GPT-4o mini"
84+
o4-mini: "o4-mini"
85+
gpt-5:
86+
search: "GPT 5"
87+
excluded_terms:
88+
[".1", ".2", ".4", ".5", ".6", "mini", "nano", "chat", "codex", "pro"]
89+
gpt-5-mini: "GPT 5 Mini"
90+
gpt-5-nano: "GPT 5 Nano"
91+
gpt-5-chat: "GPT 5 chat"
92+
gpt-5.2:
93+
search: "GPT 5.2"
94+
excluded_terms: ["pro", "chat", "ShortCo", "LongCo"]
95+
gpt-5.2-chat: "GPT 5.2 chat"
96+
6697
concurrency:
6798
max_parallel_requests: 10 # Higher than default (5) for throughput
6899
request_timeout_seconds: 120 # Longer timeout for complex prompts

‎configs/live_demo.yaml‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Live evaluation using the leestott-mr-resource Foundry deployments.
2+
# Credentials are injected into the process environment by scripts/demo.ps1.
3+
evaluation:
4+
name: "live-model-router-demo"
5+
dataset: "datasets/zava_custom.jsonl"
6+
sample_size: null
7+
random_seed: 42
8+
9+
endpoints:
10+
model_router:
11+
type: "azure_openai"
12+
endpoint_url: "${AZURE_MODEL_ROUTER_ENDPOINT}"
13+
api_key: "${AZURE_MODEL_ROUTER_KEY}"
14+
deployment_name: "${AZURE_MODEL_ROUTER_DEPLOYMENT}"
15+
parameters:
16+
max_tokens: 4096
17+
18+
baseline:
19+
type: "azure_openai"
20+
endpoint_url: "${AZURE_OPENAI_ENDPOINT}"
21+
api_key: "${AZURE_OPENAI_KEY}"
22+
deployment_name: "${AZURE_BASELINE_DEPLOYMENT}"
23+
parameters:
24+
max_tokens: 4096
25+
26+
pricing:
27+
model_router:
28+
input: 0.50
29+
output: 1.50
30+
baseline-gpt:
31+
input: 2.50
32+
output: 15.00
33+
gpt-5.4:
34+
input: 2.50
35+
output: 15.00
36+
gpt-5-mini:
37+
input: 0.25
38+
output: 2.00
39+
gpt-5-nano:
40+
input: 0.05
41+
output: 0.40
42+
# Keep this estimate aligned with the current Foundry price sheet.
43+
gpt-5.6-terra:
44+
input: 2.00
45+
output: 12.00
46+
gpt-5.6-luna:
47+
input: 0.20
48+
output: 1.20
49+
50+
pricing_source:
51+
azure_retail:
52+
enabled: true
53+
currency: "USD"
54+
region: "eastus"
55+
cache_file: ".cache/azure-retail-prices.json"
56+
cache_ttl_hours: 24
57+
timeout_seconds: 20
58+
models:
59+
gpt-5.4:
60+
search: "5.4"
61+
excluded_terms: ["pro", "mini", "nano", "pp", "ShortCo", "LongCo"]
62+
gpt-5-mini: "GPT 5 Mini"
63+
gpt-5-nano: "GPT 5 Nano"
64+
gpt-5.6-terra:
65+
search: "5.6 terra"
66+
required_terms: ["ShortCo", "Std"]
67+
gpt-5.6-luna:
68+
search: "5.6 luna"
69+
required_terms: ["ShortCo", "Std"]
70+
71+
concurrency:
72+
max_parallel_requests: 2
73+
request_timeout_seconds: 120
74+
max_retries: 3
75+
76+
output:
77+
directory: "results"
78+
formats: ["markdown", "csv", "json"]
79+
80+
judge:
81+
enabled: true
82+
endpoint:
83+
type: "azure_openai"
84+
endpoint_url: "${AZURE_JUDGE_ENDPOINT}"
85+
api_key: "${AZURE_JUDGE_KEY}"
86+
deployment_name: "${AZURE_JUDGE_DEPLOYMENT}"
87+
parameters:
88+
max_tokens: 1024
89+
pairwise_template: "configs/judge_prompts/pairwise.yaml"
90+
absolute_template: "configs/judge_prompts/absolute.yaml"
91+
max_parallel: 2
92+
timeout_seconds: 120
93+
max_retries: 3

‎docs/methodology.md‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,19 @@ The underlying model is identified from each API response's `model` field and ma
7979

8080
### Pricing source
8181

82-
All prices are in USD per 1 million tokens, configured in the YAML config under `pricing:`. The default config includes 24 models with current Azure pricing. **Update these whenever Azure pricing changes** — the tool can't fetch them automatically.
82+
Prices are expressed per 1 million tokens. When
83+
`pricing_source.azure_retail.enabled` is true, the tool queries the public
84+
[Azure Retail Prices API](https://learn.microsoft.com/rest/api/cost-management/retail-prices/azure-retail-prices)
85+
for current Foundry Models consumption meters and caches resolved prices for
86+
24 hours by default.
87+
88+
The resolver accepts only unambiguous global-standard, non-batch,
89+
non-cached, non-priority input/output token meters. Values under `pricing:`
90+
remain explicit fallbacks when the API is unavailable, a meter cannot be
91+
resolved safely, or a Marketplace model does not expose a compatible meter.
92+
Set `pricing_source.azure_retail.region` to the Foundry resource's ARM region
93+
when regional records differ. The Model Router markup remains explicitly
94+
configured.
8395

8496
## Latency measurement
8597

@@ -148,8 +160,8 @@ These are computed using Python-based graders that operate on the same cost and
148160
|--------|---------------|----------------|
149161
| Quality absolute | score ≥ 3 | Response is adequate or better |
150162
| Quality pairwise | score ≥ 3 | Router is at least as good as baseline |
151-
| Cost comparison | ratio ≥ 0.5 | Router saves ≥ 50% of baseline cost |
152-
| Latency comparison | ratio ≥ 0.5 | Router is ≥ 50% faster than baseline |
163+
| Cost comparison | score ≥ 0.5 | Router costs no more than the baseline |
164+
| Latency comparison | score ≥ 0.5 | Router is no slower than the baseline |
153165

154166
### Interpreting cross-validation
155167

0 commit comments

Comments
 (0)