token-profile compares every run against a dispatching benchmark of 75,000
cached tokens per call. No run in the retained corpus has come near that
number for days. The bar is stale, and the way it is stored guarantees it will
go stale again.
The measurements
| run |
cache read / call |
20260802T130003Z — the current benchmark |
75,000 |
20260812T162035Z |
64,563 |
20260815T110709Z |
54,532 |
Both retained traces that did real work beat the benchmark, the more recent one
by 27%. (The other 19 retained traces are SKIP ticks with no tool calls, so
they carry no per-call figure.)
Why this is not a one-line edit
BENCH_DISPATCHING_PER_CALL is not a target someone chose. It is a
measurement of one named run, and lowering the number in place would make
the record of 20260802T130003Z false. What should move is which run the
benchmark points at — re-anchor the dispatching benchmark on
20260815T110709Z at 54,532.
The inline benchmark (264,000, 20260804T114433Z) should NOT move. It is the
pathology reference and there is no newer inline run to re-anchor it on; it
stays as the record of what inline cost.
The part that will re-break
The same figure lives on three surfaces today:
pr-review-report-rs/src/main.rs:37168-37171 — BENCH_DISPATCHING_PER_CALL,
BENCH_INLINE_PER_CALL, and the two BENCH_*_RUN trace ids.
campaign-prompt.txt, as prose in the FAN OUT BY DEFAULT paragraph:
"20260802T130003Z dispatched 18 agents over 2,676 tool calls and read 75k
cached tokens per call; 20260804T114433Z did its three items inline over
414 calls and read 264k per call — 3.5x".
token-profile's own output line, which renders the constants and labels
their provenance as campaign-prompt.txt.
Three copies of one measurement, none checked against the trace they claim to
describe. Re-pointing them by hand fixes today's number and leaves the same
trap armed.
This is the defect #290 hit from the other direction: that branch asserted the
digest was "19 lines" in four places, its own change made it 21, and the fix
was to assert the prose against digest_lines(...).len() rather than to
correct four strings. Same remedy applies here.
Proposed
- Re-anchor the dispatching benchmark on
20260815T110709Z / 54,532.
- Make the benchmark derived or asserted, not restated: either compute it
from the retained corpus (the most recent dispatching run's own per-call
figure), or keep the constant and add a test that reads the named trace and
fails when the constant disagrees with it. A stale benchmark should fail a
test, not quietly misinform a run.
- Whatever the prompt says in prose must come from the same source, so the
three surfaces cannot drift apart again.
Sequencing
Blocked behind #290, which is editing both
pr-review-report-rs/src/main.rs and campaign-prompt.txt on
state-load-typed-rows-290. Doing this concurrently would collide in both
files.
token-profilecompares every run against a dispatching benchmark of 75,000cached tokens per call. No run in the retained corpus has come near that
number for days. The bar is stale, and the way it is stored guarantees it will
go stale again.
The measurements
20260802T130003Z— the current benchmark20260812T162035Z20260815T110709ZBoth retained traces that did real work beat the benchmark, the more recent one
by 27%. (The other 19 retained traces are SKIP ticks with no tool calls, so
they carry no per-call figure.)
Why this is not a one-line edit
BENCH_DISPATCHING_PER_CALLis not a target someone chose. It is ameasurement of one named run, and lowering the number in place would make
the record of
20260802T130003Zfalse. What should move is which run thebenchmark points at — re-anchor the dispatching benchmark on
20260815T110709Zat 54,532.The inline benchmark (
264,000,20260804T114433Z) should NOT move. It is thepathology reference and there is no newer inline run to re-anchor it on; it
stays as the record of what inline cost.
The part that will re-break
The same figure lives on three surfaces today:
pr-review-report-rs/src/main.rs:37168-37171—BENCH_DISPATCHING_PER_CALL,BENCH_INLINE_PER_CALL, and the twoBENCH_*_RUNtrace ids.campaign-prompt.txt, as prose in the FAN OUT BY DEFAULT paragraph:"
20260802T130003Zdispatched 18 agents over 2,676 tool calls and read 75kcached tokens per call;
20260804T114433Zdid its three items inline over414 calls and read 264k per call — 3.5x".
token-profile's own output line, which renders the constants and labelstheir provenance as
campaign-prompt.txt.Three copies of one measurement, none checked against the trace they claim to
describe. Re-pointing them by hand fixes today's number and leaves the same
trap armed.
This is the defect #290 hit from the other direction: that branch asserted the
digest was "19 lines" in four places, its own change made it 21, and the fix
was to assert the prose against
digest_lines(...).len()rather than tocorrect four strings. Same remedy applies here.
Proposed
20260815T110709Z/ 54,532.from the retained corpus (the most recent dispatching run's own per-call
figure), or keep the constant and add a test that reads the named trace and
fails when the constant disagrees with it. A stale benchmark should fail a
test, not quietly misinform a run.
three surfaces cannot drift apart again.
Sequencing
Blocked behind #290, which is editing both
pr-review-report-rs/src/main.rsandcampaign-prompt.txtonstate-load-typed-rows-290. Doing this concurrently would collide in bothfiles.