harvest(tokenaudit): est_tokens measured against a real tokenizer, and the loop the per-call number cannot see - #79
Merged
Conversation
…he loop the per-call number cannot see An outside evaluation measured ripwire spending ~10% MORE tokens than grep-and-read (callstack/agent-device #2400, 2026-09-08) and traced it to the fixed per-call legend. 8c20e10 fixed what --help SAYS about that. This adds the two instruments that would have caught it, because nothing in the tree could: est_tokens was validated for its PROPERTIES only, and its accuracy was a MAPE a human typed into a write-up in 2026-07. bench/tokenaudit/sweep.py — 25 invocations x 2 corpora, est vs tiktoken o200k_base/cl100k_base. Three findings, all in docs/EVALS.md with the numbers: - est_tokens is printed by 9 of 25 invocations. The sixteen silent ones are every navigation verb — exactly the answers whose fixed-legend share is largest. One --edit-check emitted 99,006 real tokens priced at nothing. - the signed error is +15.9%/+18.5% at the median and runs BOTH ways: -18.4% on --expand's short dense bodies, +41.7% on a budgeted --for. Real B/tok across these documents is 2.44 to 4.66 while the conversion applies one language-keyed rate near 2.5 — the error is keyed on DOCUMENT SHAPE, not on the corpus language the rate is keyed on, so no extra kTokenCalib row fixes it. - a --token-budget=3000 delivers ~2,000 real tokens, because the ceiling is a ceiling on the ESTIMATE. No constant moved. A single rate cannot correct a +40% bundle and a -16% body at once; the fix is a per-SPAN charge (prose at a prose rate, as kBytesPerTokenBody already does for bodies) and it moves a number pinned by the goldens and three budget gates. That is a round, and this is the instrument that makes its before/after measurable. test/tokenbudgetcheck.sh #18 absorbs the gate rather than adding one: it reads real counts out of test/estcalib.manifest (written out of band by bench/tokenaudit/pin.py) and holds every pin inside a measured band plus the set's MAPE under a 30% ceiling. No Python package at gate time — G3 forbids a host-installed build dependency — the same split test/printf_parity.manifest uses. The pins cannot rot: the frozen corpus test/estcalibfix is copied outside any repository and crawled by a RELATIVE path, so no at="<sha>+dirty" stamp exists to change and root="f" is one byte on every machine. Three mutation controls were run before the arm was believed: doubling one pin reddens the band, truncating the manifest reddens #18c, shrinking every pin 40% takes the MAPE to 95%. src/serialize.h §H7 said est_tokens "never systematically under-reads". Measured false on two corpora for --expand. Replaced with the range and a pointer to the gate; rounding to nearest moves half a token and the error here is a rate error worth tens of percent. bench/tokenaudit/loop_ledger.py — the whole-LOOP view, from the agent's own transcripts, aggregates only. ripwire is 42.2% of retrieval CALLS and 4.3% of retrieval TOKENS; median result 333 tokens against native's 717. And the reframing number: cache reads are 98% of everything billed, so a tool result is re-paid once per remaining turn and the per-call price sees none of it. Its Bash classifier is a deliberate port of hooks/ripwire-nudge.sh's, so a disagreement between the two is a finding — and one fired: on the 38 session ids present in both, the meter logged 4.71x the transcripts' tool calls (8.02x on native, 1.87x on ripwire). The mechanism is a subagent's PreToolUse hook reporting the PARENT's session_id, so `session` in substitution.jsonl is not the unit docs/EVALS.md §4's per-session arm reads it as. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hich half is the claim The measured magnitudes (-0.8% / -1.9%) are corpus-dependent and the self-corpus arm moved to -0.1% once this lane's own commit changed the tree it ranks. Both readings are recorded with the ref they were taken at, and the sentence now carries the DIRECTION as the claim — a wash or a small loss on --for, never the 39-66% token saving the navigation verbs show — because a number that moves with the corpus is not the thing a reader should carry away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s — ripwirepubliccheck was right
The first committed results JSON carried all three leak classes at once: an absolute home path in `bin`
and two `root` values, the private development tree's NAME as a corpus key, and the personal identifier
inside those paths. test/ripwirepubliccheck.sh caught every one on CI shard 3/4, on all four Linux
flavours, which is the gate working exactly as designed — a calibration sweep WANTS the corpora that must
not appear in a public repository, so this class was going to happen the moment the instrument was
committed rather than run.
Fixed FORWARD, per the standing directive, and fixed at the SHAPE rather than the instance (CONTRIBUTING
§2 rule 5): sweep.py no longer writes a path at all. The caller's LABEL is the corpus identity
(`root_label`), `bin` is a basename, and the file carries a `redaction` key saying so, so a reader knows
the omission is deliberate and that no measurement was altered. The committed results keep every number
exactly as taken; only the identifying strings changed, and the second corpus is now named by its shape
("a private 1500-file C++ tree"), which is how docs/EVALS.md and the README already referred to it.
Two smaller corrections in the same file: the header still named a `test/estcalibcheck.sh` that was never
created — the arm was absorbed into test/tokenbudgetcheck.sh #18 — and the new redaction comment tripped
arm 2 by quoting the very path prefix it was warning about.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (13)
Cache: Disabled due to data retention organization setting Knowledge base: Disabled due to 📝 SummarySummary by CodeRabbit
WalkthroughThe change adds token-audit measurement scripts, frozen calibration fixtures, a manifest generator, token-budget validation, benchmark results, and documentation for estimate accuracy and transcript token usage. ChangesToken audit and calibration
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant sweep.py
participant ripwire
participant tiktoken
participant ResultsJSON
Operator->>sweep.py: provide binary and corpus
sweep.py->>ripwire: derive targets and run invocations
ripwire-->>sweep.py: command output and est_tokens
sweep.py->>tiktoken: count command output
tiktoken-->>sweep.py: tokenizer counts
sweep.py->>ResultsJSON: write measurement report
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Harvest round, lane C (codeburn). Two measurement instruments, one deleted false claim, and no feature.
est_tokensmeasured against a real tokenizer for the first time. Everything that validated it before validated its properties — present, positive, deterministic, monotone under a tighter budget. Never its accuracy;tokenbudgetcheck's own header said the figure "is REPORTED by the agent in the T1 write-up", which was 2026-07 and had never been re-derived. 25 invocations × 2 corpora, real tokens counted witho200k_baseandcl100k_base(which agree to 1.4%, re-derivingkTokenCalib's ≤4% claim).Three findings:
--callers,--callees,--impact,--uses,--affected,--edit-check,--grep,--test-gate,--hotspots,--lint,--tree,--clones— and both JSON dialects. The price is missing exactly where the fixed-legend share is highest. One--edit-checkemitted 348,224 B / 99,006 real tokens in a single answer, priced at nothing and capped by nothing.kTokenCalibrow fixes it.--token-budget=Ndelivers 48–82% of N. At binding budgets the estimate over-reads 25–42%, and the budget is a hard ceiling on the estimate, so a caller asking for 3,000 tokens of context is handed about 2,000.One sentence in
src/serialize.hwas false and is gone. It said the number "never systematically under-reads". It was false on two corpora. It is replaced by the measured range and a pointer to the gate — the only source change in the lane, eighteen lines, all comment.kTokenCalibis unchanged, and that is a decision rather than an omission. The error is signed both ways and keyed on document shape: no single rate corrects a +40% legend-heavy bundle and a −16% short body at once. The change that would is a per-span charge, which moves numbers pinned by the goldens,fornotesbudgetcheck,forbudgetmonotoncheckandpacktaskquotacheck. That is its own round. What this lane leaves is the instrument that makes such a round's before/after measurable:tokenbudgetcheck#18 holds every pinned invocation inside a measured band and the set's MAPE under a 30% ceiling, against counts written intotest/estcalib.manifestfrom the frozen corpustest/estcalibfix. No host-installed dependency — the tokenizer runs out of band and the gate reads numbers, the same splitprintf_parity.manifestuses, because G3 forbids one. Three mutation controls were run before it was believed.One negative worth recording:
--legend=compactis not a saving on--for— measured −0.8% and −1.9% in tokens, because--foris budget-shaped and the bytes the legend frees are refilled from the trim ladder's tail.--help's advice is right for the navigation verbs and wrong-signed for the bundle it also names.loop_ledger.py— every number ripwire prints is per call; the claim it makes is per loop.substitution_report.py§5 counts the calls and deliberately prints no byte or token figure, so a verb could hold its terminality rate steady while its answers doubled in size and nothing would move.Squash-merged rather than merged. The lane scrubbed a private-tree leak forward in
052bab9c, so the tree is clean bute0f2ec9astill carried a home path, the private tree name and a personal identifier. Squashing keeps that out of main's history; the lane ref is deleted with it. Verified before landing: contains current main, CI run 34362431767 green 26/26 on the tip, README's gate count equals the count derived fromregression.sh's own loop (564 — unchanged, the lane adds no gate), zero private-corpus and zero personal-identifier hits.🤖 Generated with Claude Code