Skip to content

Commit 9dda14d

Browse files
committed
fix(inference): price Agentic revenue using measured cache hits
Share Fleet-compatible cache-tier aggregation, interpolate throughput, token share, and cache hit before pricing, and display uncached, cached, and output prices. Use OpenRouter input_cache_read when published with the documented fallback. 中文:复用 Fleet 的缓存层级汇总规则,在定价前分别插值吞吐量、输入 token 占比和缓存命中率,并展示未缓存输入、缓存输入及输出价格。优先使用 OpenRouter 发布的 input_cache_read,缺失时采用已说明的回退价格。
1 parent 86daa00 commit 9dda14d

27 files changed

Lines changed: 565 additions & 103 deletions

‎.claude/skills/write-inferencex-blog/iso_interactivity.py‎

Lines changed: 103 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,12 @@
2424
Two independent splines need not preserve `metric * throughput = constant` between
2525
measured knots. Pass `reciprocal_of='throughput'` (or the matching output/input
2626
throughput key) to spline that throughput and re-derive the metric. The live
27-
inference charts also expose tokens/$ and normalized token revenue as separate
28-
metrics; both are directly proportional to throughput. OpenRouter-priced revenue
29-
is proportional only when the input/output mix stays constant; otherwise spline
30-
the already-priced metric directly. Pass
31-
`proportional_to='throughput'` (or the matching output/input throughput key) so
32-
they use the same Pareto knots and spline as that throughput before applying the
33-
recovered multiplier. See docs/tco-calculator.md for a reproducible measurement.
27+
inference charts also expose tokens/$ and token revenue as separate metrics.
28+
Tokens/$ is directly proportional to throughput. For cache-aware token revenue,
29+
pass `revenue_pricing` with input/cached-input/output prices; the helper then
30+
splines throughput, measured cache hit, and input share independently before
31+
pricing the interpolated point, matching Fleet Lifecycle and Historical Trends.
32+
See docs/tco-calculator.md for a reproducible measurement.
3433
3534
Usage as a module:
3635
from iso_interactivity import interpolate_metric, pareto_front_upper_left
@@ -208,6 +207,69 @@ def recover_proportional_multiplier(
208207
return multiplier
209208

210209

210+
def measured_cache_hit_rate(point: dict) -> Optional[float]:
211+
"""GPU plus external when external exists, otherwise GPU plus CPU."""
212+
gpu = point.get('server_gpu_cache_hit_rate')
213+
external = point.get('server_external_cache_hit_rate')
214+
cpu = point.get('server_cpu_cache_hit_rate')
215+
secondary = external if isinstance(external, (int, float)) else cpu
216+
if not isinstance(gpu, (int, float)) and not isinstance(secondary, (int, float)):
217+
return None
218+
total = (float(gpu) if isinstance(gpu, (int, float)) else 0.0) + (
219+
float(secondary) if isinstance(secondary, (int, float)) else 0.0
220+
)
221+
return max(0.0, min(1.0, total))
222+
223+
224+
def input_token_share_for_revenue(point: dict, tput_key: str) -> Optional[float]:
225+
"""Port of inputTokenShareForRevenue in token-revenue.ts."""
226+
total = point.get('tput_per_gpu', point.get(tput_key, 0))
227+
input_tput = point.get('input_tput_per_gpu', 0)
228+
output_tput = point.get('output_tput_per_gpu', 0)
229+
if all(isinstance(value, (int, float)) for value in (total, input_tput, output_tput)):
230+
stream_sum = float(input_tput) + float(output_tput)
231+
if float(total) > 0 and stream_sum > 0 and abs(stream_sum / float(total) - 1) <= 0.01:
232+
return float(input_tput) / stream_sum
233+
234+
isl = point.get('isl')
235+
osl = point.get('osl')
236+
if isinstance(isl, (int, float)) and isinstance(osl, (int, float)) and isl + osl > 0:
237+
return float(isl) / float(isl + osl)
238+
239+
prompt = point.get('total_prompt_tokens')
240+
generation = point.get('total_generation_tokens')
241+
if (
242+
isinstance(prompt, (int, float))
243+
and isinstance(generation, (int, float))
244+
and prompt + generation > 0
245+
):
246+
return float(prompt) / float(prompt + generation)
247+
return None
248+
249+
250+
def token_revenue_from_rates_per_gpu_hour(
251+
total_tput: float,
252+
input_share: Optional[float],
253+
cache_hit_rate: Optional[float],
254+
pricing: dict,
255+
) -> Optional[float]:
256+
"""Port of tokenRevenueFromRatesPerGpuHour in token-revenue.ts."""
257+
if not total_tput > 0:
258+
return 0.0
259+
input_price = float(pricing['input_per_million'])
260+
output_price = float(pricing['output_per_million'])
261+
cached_price = float(pricing.get('cached_input_per_million', input_price * 0.1))
262+
hit = max(0.0, min(1.0, cache_hit_rate)) if cache_hit_rate is not None else 0.0
263+
if input_price == output_price and not (hit > 0 and cached_price != input_price):
264+
return total_tput * 3600 * input_price / 1_000_000
265+
if input_share is None:
266+
return None
267+
share = max(0.0, min(1.0, input_share))
268+
input_price_at_point = (1 - hit) * input_price + hit * cached_price
269+
blended_price = share * input_price_at_point + (1 - share) * output_price
270+
return total_tput * 3600 * blended_price / 1_000_000
271+
272+
211273
def interpolate_metric(
212274
points: list[dict],
213275
target_iv: float,
@@ -216,6 +278,7 @@ def interpolate_metric(
216278
tput_key: str = 'throughput',
217279
reciprocal_of: Optional[str] = None,
218280
proportional_to: Optional[str] = None,
281+
revenue_pricing: Optional[dict] = None,
219282
) -> Optional[float]:
220283
"""Interpolate `metric_key` at `target_iv` using the chart's algorithm.
221284
@@ -240,6 +303,10 @@ def interpolate_metric(
240303
with a constant input/output mix). That
241304
throughput is splined and the recovered multiplier applied, matching the
242305
dashboard.
306+
307+
`revenue_pricing` switches token revenue to the Fleet-compatible path:
308+
interpolate throughput, input share, and cache hit independently, then price
309+
that operating point. A partly measured cache frontier receives no discount.
243310
"""
244311
if not points:
245312
return None
@@ -275,6 +342,32 @@ def interpolate_metric(
275342
return None
276343
ys.append(value)
277344

345+
if revenue_pricing is not None:
346+
def interpolate_bounded(values: list[float]) -> float:
347+
slopes = monotone_slopes(xs, values)
348+
raw = hermite_interpolate(xs, values, slopes, target_iv)
349+
return max(min(values), min(max(values), raw))
350+
351+
revenue_tput_key = proportional_to or tput_key
352+
tputs = [point.get(revenue_tput_key) for point in sorted_front]
353+
if any(value is None for value in tputs):
354+
return None
355+
input_shares = [
356+
input_token_share_for_revenue(point, revenue_tput_key) for point in sorted_front
357+
]
358+
cache_hit_rates = [measured_cache_hit_rate(point) for point in sorted_front]
359+
input_share = (
360+
interpolate_bounded(input_shares) if all(v is not None for v in input_shares) else None
361+
)
362+
cache_hit_rate = (
363+
interpolate_bounded(cache_hit_rates)
364+
if all(v is not None for v in cache_hit_rates)
365+
else None
366+
)
367+
return token_revenue_from_rates_per_gpu_hour(
368+
interpolate_bounded(tputs), input_share, cache_hit_rate, revenue_pricing
369+
)
370+
278371
if proportional_to is not None:
279372
tputs: list[float] = []
280373
for point in sorted_front:
@@ -324,7 +417,8 @@ def interpolate_metric(
324417
def _cli() -> None:
325418
"""Stdin: {"points": [...], "target_iv": N, "metric_key": "...",
326419
"reciprocal_of": "throughput" for $/M tok and J/token,
327-
"proportional_to": "throughput" for tokens/$ or revenue with a constant mix}
420+
"proportional_to": "throughput" for tokens/$,
421+
"revenue_pricing": {...} for cache-aware revenue}
328422
Stdout: {"value": N or null}"""
329423
req = json.loads(sys.stdin.read())
330424
value = interpolate_metric(
@@ -335,6 +429,7 @@ def _cli() -> None:
335429
tput_key=req.get('tput_key', 'throughput'),
336430
reciprocal_of=req.get('reciprocal_of'),
337431
proportional_to=req.get('proportional_to'),
432+
revenue_pricing=req.get('revenue_pricing'),
338433
)
339434
json.dump({'value': value}, sys.stdout)
340435
sys.stdout.write('\n')

‎docs/data-transforms.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ Returns `{ chartData: InferenceData[][], hardwareConfig: HardwareConfig }`.
6060
**`buildDerivedChartFields(entry, hwKey, requestedMetrics?)`** is the sole owner of inference and historical trend formulas. The full transform requests every benchmark-backed field. History requests only the selected metric and its interpolation dependencies, so it stays lightweight. Each emitted value uses `{ y: number; roof: boolean }`.
6161

6262
- `tpPerGpu`, `outputTputPerGpu`, `inputTputPerGpu` — raw throughput from the entry (tok/s/gpu).
63-
- `tokenRevenuePerGpuHour` — gross revenue in $/GPU/hr. `buildDerivedChartFields` emits the normalized default (`tputPerGpu × 3,600 ÷ 1,000,000`, with input and output both priced at $1/M). `applyTokenRevenuePricing` can replace it with the selected model's current public OpenRouter prompt/completion prices. Aggregate rows use their measured input/output split; disaggregated rows derive a like-for-like split from fixed ISL:OSL or measured agentic prompt:generation totals because their raw input/output rates use different prefill/decode GPU denominators.
63+
- `tokenRevenuePerGpuHour` — gross revenue in $/GPU/hr. `buildDerivedChartFields` emits an uncached normalized placeholder; `applyTokenRevenuePricing` applies the selected normalized or OpenRouter prices to uncached input, cached input, and output. Normalized pricing uses $1/M for uncached input and output and $0.10/M for cached input. OpenRouter uses its published cache-read price when available, otherwise 10% of the prompt price. Agentic cache share uses the same measured GPU-plus-external-or-CPU rule as Fleet Lifecycle. Aggregate rows use their measured input/output split; disaggregated rows derive a like-for-like split from fixed ISL:OSL or measured agentic prompt:generation totals because their raw input/output rates use different prefill/decode GPU denominators. Historical Trends splines total throughput, token share, and cache hit independently before applying these prices; a partly measured cache frontier receives no cache discount.
6464
- `tpPerMw` — `(tputPerGpu * 1000) / hardwarePower` (GPU power in kW, result in tok/s/MW).
6565
- Cost-per-million fields — GPU hourly cost divided by tokens-per-hour (in millions): `costh` / `costn` / `costr` for hyperscaler / neocloud / 3-year-rental pricing respectively. Three token variants exist: combined (`costh`/`costn`/`costr`), output-only (`costhOutput`/`costnOutput`/`costrOutput`), and input-only (`costhi`/`costni`/`costri`).
6666
- Tokens-per-dollar fields — tokens-per-hour divided by the same hourly costs. They are separate Y-axis metrics rather than a display toggle: combined (`tokensPerDollarH`/`N`/`R`), output-only (`outputTokensPerDollarH`/`N`/`R`), and input-only (`inputTokensPerDollarH`/`N`/`R`). This keeps existing `i_metric=y_cost*` links and cost semantics unchanged.

‎docs/state-ownership.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ them.
113113
Axis and presentation state:
114114

115115
- selected x-axis and y-axis metrics, percentile, and effective x-axis mode
116-
- token-revenue price source (`i_revenue`): normalized $1/M or the selected model's live OpenRouter catalog prices
116+
- token-revenue price source (`i_revenue`): normalized uncached/cached/output pricing or the selected model's live OpenRouter catalog prices
117117
- scale, optimal-point, label, contrast, legend, and overlay controls
118118

119119
Display changes stay in this domain. A contrast or label toggle therefore does not

‎docs/tco-calculator.md‎

Lines changed: 34 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -167,27 +167,39 @@ at every interpolated point.
167167
compare interpolation models on a fixed snapshot, but they must not be presented
168168
as permanent impact figures for the changing live dataset.
169169

170-
The `/inference` page also exposes tokens-per-dollar and normalized token revenue
171-
as separate Y-axis metrics; neither replaces the cost-per-million metrics.
172-
Historical trends for these proportional metrics select Pareto knots from the
173-
matching throughput, spline that throughput, and apply the constant multiplier.
174-
Tokens-per-dollar uses `3600 / $/GPU-hr`; normalized token revenue uses
175-
`3600 / 1,000,000` at the axis's explicit `$1/M tok` sale price. Reusing the
176-
matching throughput frontier is essential: total-throughput knots are not
177-
necessarily the output- or input-throughput Pareto knots.
178-
179-
The revenue axis defaults to pricing every input and output token at the same
180-
`$1/M tok` rate, making it a model-independent SLA comparison. Its OpenRouter
181-
option fetches the selected model's current public prompt/completion prices from
182-
`https://openrouter.ai/api/v1/models` and prices the streams separately. Aggregate
183-
rows use their measured input/output split. Disaggregated rows cannot use the raw
184-
per-prefill/per-decode rates together, so they apply the fixed ISL:OSL shape or
185-
the measured agentic prompt:generation mix to total tok/s/GPU. The OpenRouter
186-
option uses standard prompt/completion prices and does not apply cache discounts.
170+
The `/inference` page also exposes tokens-per-dollar and token revenue as
171+
separate Y-axis metrics; neither replaces the cost-per-million metrics.
172+
Historical tokens-per-dollar trends select Pareto knots from the matching
173+
throughput, spline that throughput, and apply the constant `3600 / $/GPU-hr`
174+
multiplier. Revenue selects the total-throughput Pareto knots, interpolates their
175+
throughput, token mix, and cache hit independently, and then applies the selected
176+
prices. Reusing the matching frontier remains essential: total-throughput knots
177+
are not necessarily the output- or input-throughput Pareto knots.
178+
179+
The revenue axis defaults to normalized cache-aware pricing: uncached input and
180+
output are `$1/M tok`, while cached input is `$0.10/M tok`, matching Fleet
181+
Lifecycle's default cache-read assumption. Its OpenRouter option fetches the
182+
selected model's current public prompt/completion/cache-read prices from
183+
`https://openrouter.ai/api/v1/models`; when the catalog omits a cache-read price,
184+
the same 10%-of-input fallback is used and the resulting price is printed in the
185+
plot subtitle. Aggregate rows use their measured input/output split.
186+
Disaggregated rows cannot use the raw per-prefill/per-decode rates together, so
187+
they apply the fixed ISL:OSL shape or the measured agentic prompt:generation mix
188+
to total tok/s/GPU.
189+
190+
Agentic points use the same measured cache-tier rule as Fleet Lifecycle: GPU plus
191+
external when the external metric is present, otherwise GPU plus CPU, clamped to
192+
`[0,1]`. Missing cache telemetry bills the input stream at the uncached price.
193+
Historical Trends uses the same component-wise calculation as Fleet Lifecycle:
194+
it splines total throughput, input-token share, and measured cache hit separately
195+
on the total-throughput frontier, clamps each to its measured range, and only then
196+
applies the selected prices. A frontier with cache telemetry on only some knots
197+
opts out of the cache discount instead of inventing zero-hit measurements.
187198

188199
Neither mode is the fleet lifecycle section's realized-revenue model: it does not
189-
model availability, rollout, or a user-supplied cached-input discount. Those
190-
business assumptions remain in Fleet Lifecycle below.
200+
model availability or rollout, and its cached price comes from the selected price
201+
source rather than Fleet Lifecycle's user-editable assumption. Those business
202+
assumptions remain in Fleet Lifecycle below.
191203

192204
### The consistency guard
193205

@@ -854,8 +866,8 @@ Three properties are worth stating, because they are what make this safe:
854866
- **A partly-measured frontier opts out.** `interpolateForGPU` splines the rate
855867
only when _every_ frontier point carries one; substituting 0 for the missing
856868
points would invent a dip in the cached fraction and overstate the billable
857-
rate. Opting out bills every input token at full price instead — wrong in the
858-
direction that understates margin.
869+
rate. Opting out bills every input token at full price instead, which
870+
overstates revenue and margin when the missing point actually had cache hits.
859871

860872
`cacheReadRatio` is the one user input here, labelled `Cached input (% of price)`
861873
and defaulting to 10% (the ratio DeepSeek and Anthropic both publish). The cached
@@ -879,7 +891,7 @@ The middle two rows carry the decision. Where an external rate is reported,
879891
adding CPU on top breaches the ceiling 56 times out of 106 — the router-side
880892
external figure already contains the offload tier, so summing double-counts it.
881893
Where no external rate is reported the CPU tier is real and disjoint: 0 breaches
882-
across all 26, every one of them an offload-on row. Hence `cacheHitRateOf` adds
894+
across all 26, every one of them an offload-on row. Hence `measuredCacheHitRate` adds
883895
external when present and CPU only in its absence.
884896

885897
Dropping the CPU tier outright, which this did until 2026-08-19, understated the

‎packages/app/cypress/e2e/inference-chart.cy.ts‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ describe('Inference Chart', () => {
8484
id: 'deepseek/deepseek-v4-pro-0813',
8585
pricing: {
8686
prompt: '0.000001122',
87+
input_cache_read: '0.00000008',
8788
completion: '0.000003366',
8889
},
8990
},
@@ -112,12 +113,16 @@ describe('Inference Chart', () => {
112113
'OpenRouter current pricing',
113114
);
114115
cy.get('[data-testid="openrouter-price-summary"]')
115-
.should('contain.text', 'Input $1.122/M tok')
116+
.should('contain.text', 'Uncached input $1.122/M tok')
117+
.and('contain.text', 'Cached input $0.08/M tok')
116118
.and('contain.text', 'Output $3.366/M tok');
117119
cy.get('[data-testid="chart-figure"]')
118120
.first()
119121
.find('[data-testid="token-revenue-subtitle-prices"]')
120-
.should('have.text', 'Input $1.122/M tok · Output $3.366/M tok')
122+
.should(
123+
'have.text',
124+
'Uncached input $1.122/M tok · Cached input $0.08/M tok · Output $3.366/M tok',
125+
)
121126
.then(($prices) => {
122127
const subtitle = $prices.parent().text();
123128
expect(subtitle.indexOf($prices.text())).to.be.lessThan(subtitle.indexOf('Updated:'));
@@ -156,6 +161,7 @@ describe('Inference Chart', () => {
156161
id: 'deepseek/deepseek-v4-pro-0813',
157162
pricing: {
158163
prompt: '0.000001122',
164+
input_cache_read: '0.00000008',
159165
completion: '0.000003366',
160166
},
161167
},
@@ -178,12 +184,16 @@ describe('Inference Chart', () => {
178184
'OpenRouter 当前价格',
179185
);
180186
cy.get('[data-testid="openrouter-price-summary"]')
181-
.should('contain.text', '输入 $1.122/百万 token')
187+
.should('contain.text', '未缓存输入 $1.122/百万 token')
188+
.and('contain.text', '缓存输入 $0.08/百万 token')
182189
.and('contain.text', '输出 $3.366/百万 token');
183190
cy.get('[data-testid="chart-figure"]')
184191
.first()
185192
.find('[data-testid="token-revenue-subtitle-prices"]')
186-
.should('have.text', '输入 $1.122/百万 token · 输出 $3.366/百万 token')
193+
.should(
194+
'have.text',
195+
'未缓存输入 $1.122/百万 token · 缓存输入 $0.08/百万 token · 输出 $3.366/百万 token',
196+
)
187197
.then(($prices) => {
188198
const subtitle = $prices.parent().text();
189199
expect(subtitle.indexOf($prices.text())).to.be.lessThan(subtitle.indexOf('更新时间:'));

‎packages/app/cypress/support/overlay-fixtures.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,10 @@ export const metricsFor = (intvty: number, tput: number, e2el: number): Record<s
4040
tput_per_gpu: tput,
4141
output_tput_per_gpu: tput * 0.3,
4242
input_tput_per_gpu: tput * 0.7,
43+
server_gpu_cache_hit_rate: 0.8,
44+
server_external_cache_hit_rate: 0.1,
45+
// External already includes the offload tier; revenue must not add this again.
46+
server_cpu_cache_hit_rate: 0.05,
4347
});
4448

4549
/**

‎packages/app/src/components/calculator/FleetLifecycle.tsx‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ import { SegmentedToggle, type SegmentedToggleOption } from '@/components/ui/seg
2727
import { TooltipProvider } from '@/components/ui/tooltip';
2828
import { track } from '@/lib/analytics';
2929
import { exportToCsv } from '@/lib/csv-export';
30+
import { DEFAULT_CACHED_INPUT_PRICE_RATIO } from '@/lib/cache-pricing';
3031
import { getGpuSpecs, getHardwareConfig } from '@/lib/constants';
3132
import {
3233
getModelLabel,
@@ -343,7 +344,7 @@ const DEFAULTS = {
343344
* and Anthropic both publish, and the order of magnitude the others sit at.
344345
* An assumption like the rest; the cached *fraction* it applies to is measured.
345346
*/
346-
cachedInputPct: 10,
347+
cachedInputPct: DEFAULT_CACHED_INPUT_PRICE_RATIO * 100,
347348
/**
348349
* An output token sells for four times an input one until the user says
349350
* otherwise — DeepSeek's own published API pricing is $0.27 / $1.10, and the

0 commit comments

Comments
 (0)