Skip to content

token-profile's dispatching benchmark is stale, and stored so it will go stale again #291

Description

@thedavidmeister

token-profile compares every run against a dispatching benchmark of 75,000
cached tokens per call
. No run in the retained corpus has come near that
number for days. The bar is stale, and the way it is stored guarantees it will
go stale again.

The measurements

run cache read / call
20260802T130003Z — the current benchmark 75,000
20260812T162035Z 64,563
20260815T110709Z 54,532

Both retained traces that did real work beat the benchmark, the more recent one
by 27%. (The other 19 retained traces are SKIP ticks with no tool calls, so
they carry no per-call figure.)

Why this is not a one-line edit

BENCH_DISPATCHING_PER_CALL is not a target someone chose. It is a
measurement of one named run, and lowering the number in place would make
the record of 20260802T130003Z false. What should move is which run the
benchmark points at — re-anchor the dispatching benchmark on
20260815T110709Z at 54,532.

The inline benchmark (264,000, 20260804T114433Z) should NOT move. It is the
pathology reference and there is no newer inline run to re-anchor it on; it
stays as the record of what inline cost.

The part that will re-break

The same figure lives on three surfaces today:

  1. pr-review-report-rs/src/main.rs:37168-37171BENCH_DISPATCHING_PER_CALL,
    BENCH_INLINE_PER_CALL, and the two BENCH_*_RUN trace ids.
  2. campaign-prompt.txt, as prose in the FAN OUT BY DEFAULT paragraph:
    "20260802T130003Z dispatched 18 agents over 2,676 tool calls and read 75k
    cached tokens per call; 20260804T114433Z did its three items inline over
    414 calls and read 264k per call — 3.5x".
  3. token-profile's own output line, which renders the constants and labels
    their provenance as campaign-prompt.txt.

Three copies of one measurement, none checked against the trace they claim to
describe. Re-pointing them by hand fixes today's number and leaves the same
trap armed.

This is the defect #290 hit from the other direction: that branch asserted the
digest was "19 lines" in four places, its own change made it 21, and the fix
was to assert the prose against digest_lines(...).len() rather than to
correct four strings. Same remedy applies here.

Proposed

  • Re-anchor the dispatching benchmark on 20260815T110709Z / 54,532.
  • Make the benchmark derived or asserted, not restated: either compute it
    from the retained corpus (the most recent dispatching run's own per-call
    figure), or keep the constant and add a test that reads the named trace and
    fails when the constant disagrees with it. A stale benchmark should fail a
    test, not quietly misinform a run.
  • Whatever the prompt says in prose must come from the same source, so the
    three surfaces cannot drift apart again.

Sequencing

Blocked behind #290, which is editing both
pr-review-report-rs/src/main.rs and campaign-prompt.txt on
state-load-typed-rows-290. Doing this concurrently would collide in both
files.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions