From e011bc00308ce27088dde77c20b4d3a2c6cb6e60 Mon Sep 17 00:00:00 2001 From: joyful-ii-V-I Date: Wed, 9 Sep 2026 08:53:49 -0400 Subject: [PATCH 1/3] feat(tokenaudit): est_tokens measured against a real tokenizer, and the loop the per-call number cannot see MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An outside evaluation measured ripwire spending ~10% MORE tokens than grep-and-read (callstack/agent-device #2400, 2026-09-08) and traced it to the fixed per-call legend. 8c20e108 fixed what --help SAYS about that. This adds the two instruments that would have caught it, because nothing in the tree could: est_tokens was validated for its PROPERTIES only, and its accuracy was a MAPE a human typed into a write-up in 2026-07. bench/tokenaudit/sweep.py — 25 invocations x 2 corpora, est vs tiktoken o200k_base/cl100k_base. Three findings, all in docs/EVALS.md with the numbers: - est_tokens is printed by 9 of 25 invocations. The sixteen silent ones are every navigation verb — exactly the answers whose fixed-legend share is largest. One --edit-check emitted 99,006 real tokens priced at nothing. - the signed error is +15.9%/+18.5% at the median and runs BOTH ways: -18.4% on --expand's short dense bodies, +41.7% on a budgeted --for. Real B/tok across these documents is 2.44 to 4.66 while the conversion applies one language-keyed rate near 2.5 — the error is keyed on DOCUMENT SHAPE, not on the corpus language the rate is keyed on, so no extra kTokenCalib row fixes it. - a --token-budget=3000 delivers ~2,000 real tokens, because the ceiling is a ceiling on the ESTIMATE. No constant moved. A single rate cannot correct a +40% bundle and a -16% body at once; the fix is a per-SPAN charge (prose at a prose rate, as kBytesPerTokenBody already does for bodies) and it moves a number pinned by the goldens and three budget gates. That is a round, and this is the instrument that makes its before/after measurable. test/tokenbudgetcheck.sh #18 absorbs the gate rather than adding one: it reads real counts out of test/estcalib.manifest (written out of band by bench/tokenaudit/pin.py) and holds every pin inside a measured band plus the set's MAPE under a 30% ceiling. No Python package at gate time — G3 forbids a host-installed build dependency — the same split test/printf_parity.manifest uses. The pins cannot rot: the frozen corpus test/estcalibfix is copied outside any repository and crawled by a RELATIVE path, so no at="+dirty" stamp exists to change and root="f" is one byte on every machine. Three mutation controls were run before the arm was believed: doubling one pin reddens the band, truncating the manifest reddens #18c, shrinking every pin 40% takes the MAPE to 95%. src/serialize.h §H7 said est_tokens "never systematically under-reads". Measured false on two corpora for --expand. Replaced with the range and a pointer to the gate; rounding to nearest moves half a token and the error here is a rate error worth tens of percent. bench/tokenaudit/loop_ledger.py — the whole-LOOP view, from the agent's own transcripts, aggregates only. ripwire is 42.2% of retrieval CALLS and 4.3% of retrieval TOKENS; median result 333 tokens against native's 717. And the reframing number: cache reads are 98% of everything billed, so a tool result is re-paid once per remaining turn and the per-call price sees none of it. Its Bash classifier is a deliberate port of hooks/ripwire-nudge.sh's, so a disagreement between the two is a finding — and one fired: on the 38 session ids present in both, the meter logged 4.71x the transcripts' tool calls (8.02x on native, 1.87x on ripwire). The mechanism is a subagent's PreToolUse hook reporting the PARENT's session_id, so `session` in substitution.jsonl is not the unit docs/EVALS.md §4's per-session arm reads it as. Co-Authored-By: Claude Opus 5 --- bench/tokenaudit/README.md | 114 +++ bench/tokenaudit/loop_ledger.py | 290 ++++++ bench/tokenaudit/pin.py | 98 ++ .../results/loopledger-2026-09-09.json | 33 + .../results/tokenaudit-2026-09-09.json | 906 ++++++++++++++++++ bench/tokenaudit/sweep.py | 195 ++++ docs/EVALS.md | 132 +++ src/serialize.h | 18 +- test/estcalib.manifest | 12 + test/estcalibfix/ledger.cpp | 86 ++ test/estcalibfix/rates.py | 40 + test/estcalibfix/session.ts | 46 + test/tokenbudgetcheck.sh | 81 ++ 13 files changed, 2049 insertions(+), 2 deletions(-) create mode 100644 bench/tokenaudit/README.md create mode 100644 bench/tokenaudit/loop_ledger.py create mode 100644 bench/tokenaudit/pin.py create mode 100644 bench/tokenaudit/results/loopledger-2026-09-09.json create mode 100644 bench/tokenaudit/results/tokenaudit-2026-09-09.json create mode 100644 bench/tokenaudit/sweep.py create mode 100644 test/estcalib.manifest create mode 100644 test/estcalibfix/ledger.cpp create mode 100644 test/estcalibfix/rates.py create mode 100644 test/estcalibfix/session.ts diff --git a/bench/tokenaudit/README.md b/bench/tokenaudit/README.md new file mode 100644 index 00000000..cbaac67d --- /dev/null +++ b/bench/tokenaudit/README.md @@ -0,0 +1,114 @@ +# tokenaudit — is `est_tokens=` true? + +`est_tokens=` is the number `--token-budget` gates on, the number `--max-tokens` fits to, and the number +the tool prints when it says what an answer cost. Until 2026-09-09 nothing in the tree compared it to a +real tokenizer. `test/tokenbudgetcheck.sh` validated its *properties* — present, positive, deterministic, +monotone under a tighter budget, bounded by an allowance derived from the estimate's own constants — and +its own header said the accuracy number "is REPORTED by the agent in the T1 write-up". That write-up was +2026-07. METHODOLOGY §9 principle 6 says measuring gets its own instrument; this directory is it. + +Two artifacts, deliberately split by whether they may need a Python package: + +| File | Needs tiktoken | What it is | +| --- | --- | --- | +| `sweep.py` | yes | the survey: ~25 invocations × N corpora, est vs real, per verb, plus the legend's share in real tokens | +| `pin.py` | yes | writes `test/estcalib.manifest` — real o200k/cl100k counts on the frozen fixture `test/estcalibfix` | +| `test/tokenbudgetcheck.sh` #18 | **no** | the gate: reads the manifest, holds every pin inside a band and the set's MAPE under a ceiling | + +The split is the point. G3 is one deterministic build step with nothing host-installed, so a gate that +imported tiktoken would be a dependency the build contract forbids. The tokenizer runs by hand and writes +numbers; the gate reads numbers. Same shape as `test/printf_parity.manifest`. + +```bash +python3 -m venv /tmp/tokvenv && /tmp/tokvenv/bin/pip install tiktoken +/tmp/tokvenv/bin/python bench/tokenaudit/sweep.py --bin build/ripwire \ + --corpus self=. --corpus other=/path/to/another/tree \ + --out bench/tokenaudit/results/tokenaudit-YYYY-MM-DD.json +/tmp/tokvenv/bin/python bench/tokenaudit/pin.py --bin build/ripwire # regenerates the manifest +python3 -c "import json,statistics as s; d=json.load(open('bench/tokenaudit/results/tokenaudit-2026-09-09.json'))" +``` + +## What the 2026-09-09 run measured + +`results/tokenaudit-2026-09-09.json`: 25 invocations on this repository and on a 1,500-file private C++ +tree, `ripwire 0.5.0 built_from=4c10be9d7`, tiktoken `o200k_base` (and `cl100k_base` beside it — the two +agree to within 1.4% on every row, which is the ≤4% spread `kTokenCalib`'s header claims, re-derived). +No Anthropic `count_tokens` arm: `ANTHROPIC_API_KEY` was not in the environment, so **Claude's own +tokenizer is unmeasured here** and o200k_base remains the public stand-in the table was calibrated against. + +**1. `est_tokens=` is printed by 9 of the 25 invocations.** The sixteen that print no price include every +navigation verb — `--callers`, `--callees`, `--impact`, `--uses`, `--affected`, `--edit-check`, `--grep`, +`--test-gate`, `--hotspots`, `--lint`, `--tree`, `--clones` — and both JSON dialects. These are exactly the +answers whose legend share is largest, so the price is missing where it is highest. A `--edit-check` on a +macro with thousands of call sites emitted **348,224 B / 99,006 real tokens in one answer**, priced at +nothing. + +**2. The signed error is not centred and is not one-directional.** Per-verb, against o200k: + +| verb | this repo | 1500-file C++ tree | fixture pin | +| --- | --- | --- | --- | +| map | +2.8% | +9.8% | +11.3% | +| map `--top-k=10` | +15.9% | +19.0% | +21.4% (`--top-k=5`) | +| `--metrics` | +4.4% | +7.2% | +5.4% | +| `--pack-signatures` | +17.0% | +18.5% | +17.5% | +| `--expand` | +1.9% | **−18.4%** | **−16.4%** | +| `--for` (named) | +26.1% | +35.6% | +33.1% | +| `--for` (conceptual) | +21.9% | +25.5% | — | +| `--pack-task` | +18.9% | +24.6% | +25.2% | +| `--around` | +14.7% | +13.7% | — | + +Median +15.9% / +18.5%; MAPE over the eight pins 21%. The mechanism is measured, not guessed: real +bytes-per-token ranges **2.44 (dense signature rows) to 4.66 (legend prose)** across these documents, +while the conversion uses one language-keyed rate near 2.5 for markup and 3.8 for bodies. The error is a +property of the **document shape**, not of the corpus language the rate is keyed on — which is why adding +a language row cannot fix it, and why the `--expand` body rate that is right on large C++ bodies +under-reads by 16–18% on short dense ones. + +**3. The legend's price, in the unit the owner mandated.** `--help` states the compact saving in bytes +("at least 50% of a small `--callers`/`--uses`/`--impact`/`--affected` answer") and +`test/legendcostcheck.sh` holds the binary to it in bytes. In **tokens** the same measurement on the +gate's own symbol (`lookupLang`) reads lower on all four, because the legend is prose (4.4 B/tok) and the +rows it is compared against are markup (2.7 B/tok): + +| verb | byte saving | token saving | gap | +| --- | --- | --- | --- | +| `--callers` | 70.9% | 60.5% | 10.4 pt | +| `--uses` | 65.8% | 55.2% | 10.6 pt | +| `--impact` | 51.9% | **39.4%** | 12.5 pt | +| `--affected` | 70.1% | 65.8% | 4.3 pt | + +`--impact` clears the published 50% floor in bytes and misses it in tokens. Across the whole sweep the +legend's token share ran 3.2% (whole map) to **79.8% (`--callees`)**, with `--edit-check` 62.9% and +`--test-gate` 52.4% on this repository — corroborating the magnitude of the outside evaluation that +started this (callstack/agent-device #2400, "a fixed per-call preamble, 62% of `--callers`' whole +response") while placing `--callers` itself at 42.9%/33.3% on the two corpora here. + +**4. `--legend=compact` is not a saving on `--for`.** Measured −0.8% (this repo) and −1.9% (the C++ tree): +the compact posture emitted *more* tokens, because `--for` is budget-shaped and the bytes the legend frees +are refilled from the trim ladder's tail. `--help`'s advice ("MAKING REPEATED CALLS? USE compact") is right +for the navigation verbs and wrong-signed for the bundle it also names as "a little". + +**5. What a `--token-budget=N` actually delivers.** Real o200k tokens as a fraction of the requested N, +and the reported `est_tokens` error at that budget: + +| verb | N=1500 | N=3000 | N=6000 | +| --- | --- | --- | --- | +| `--for` delivered | 76% / 75% | 74% / 67% | 54% / 48% | +| `--for` est error | +38% / +42% | +25% / +27% | +22% / +26% | +| `--pack-task` delivered | 82% / 81% | 52% / 61% | 58% / 57% | + +(this repo / the C++ tree). Part of the shortfall at a large N is content exhaustion — there is no more to +serve — but at the binding budgets the estimate over-reads by a quarter to two fifths, and the budget is a +hard ceiling on the estimate, so a caller asking for 3,000 tokens of context is handed about 2,000. + +## What was NOT changed, and why + +No constant in `kTokenCalib` moved. The error is per-document-shape and signed both ways: no single rate, +and no per-language row, corrects a +40% on a legend-heavy bundle and a −16% on a short body at the same +time. The change that would is a per-SPAN charge (prose bytes at a prose rate, the way `kBytesPerTokenBody` +already charges body bytes at a body rate) — a behaviour change to a number pinned by goldens, +`fornotesbudgetcheck`, `forbudgetmonotoncheck`, `packtaskquotacheck` and published budget figures. That is +a round, not a lane. What this round leaves behind is the instrument that makes the round's before/after +measurable, and one corrected sentence in `src/serialize.h` — "the number never systematically +under-reads" was false for `--expand` on two corpora and is now the measured range with the gate that +holds it. diff --git a/bench/tokenaudit/loop_ledger.py b/bench/tokenaudit/loop_ledger.py new file mode 100644 index 00000000..a79a1a68 --- /dev/null +++ b/bench/tokenaudit/loop_ledger.py @@ -0,0 +1,290 @@ +#!/usr/bin/env python3 +# loop_ledger.py — what a ripwire call costs across a WHOLE EDIT LOOP, from the agent's own transcript. +# +# THE GAP. Every number ripwire prints is per call: est_tokens on the answer it just wrote. The claim the +# tool makes is per LOOP — "the agent needs no further native read after the call" (METHODOLOGY §9 +# principle 1). Those are different quantities, and only the second one can go negative: an answer that +# costs 1,400 tokens and saves three 2,000-token file reads is a win; the same answer followed by the +# three reads anyway is a 1,400-token loss. Nothing in the tree measured the second quantity in TOKENS. +# `bench/substitution_report.py` counts the CALLS (the meter's terminality section, §5) and is the +# ancestor of this file; it deliberately prints no byte or token figure, so a verb could hold its +# terminality rate steady while its answers doubled in size and the report would not move. +# +# WHAT THIS READS, AND THE PRIVACY RULE. Claude Code writes one JSONL per session under +# ~/.claude/projects//. This script reads ONE project's directory, named on the command +# line, and prints AGGREGATES: per-session token counts and ratios, and distribution summaries over +# sessions. It never prints a prompt, a file path from a transcript, a tool argument, or a session id +# unless --sessions is passed (a local-only debugging aid). Nothing it prints is intended to be pasted +# anywhere that the transcripts themselves would not be. +# +# WHAT IT COUNTS. +# * TOOL-RESULT tokens, split ripwire / native / other. This is context an answer PUT INTO the window. +# ripwire = mcp__ripwire__* tools, plus Bash commands that invoke the binary (`ripwire`, +# `build/ripwire`, `./build/ripwire`, `asan/ripwire`). +# native = Read / Grep / Glob / NotebookRead, plus Bash commands whose head is a retrieval command +# (grep, rg, cat, head, sed -n, find, ls, awk-over-a-file). These are the calls ripwire is +# a substitute FOR — the same family split bench/substitution_report.py uses. +# other = everything else (edits, builds, git, gates, agent plumbing). Counted, never in a ratio. +# * NON-TERMINAL ripwire calls: a ripwire call with at least one native call in the NEXT 3 tool calls. +# Three is the substitution meter's own definition of a post-call sweep, kept identical on purpose so +# the two instruments are comparable; the meter's own window is 5 and is reported beside it. +# * PROVIDER-REPORTED session usage from `message.usage` (input / output / cache_creation / +# cache_read). This is the codeburn-style measurement: what the API actually billed, not an estimate. +# It is reported beside the tool-result totals so the tool-result share of a session is visible. +# +# TOKENIZER. tiktoken if importable (o200k_base), else bytes / 2.5 with `tokenizer=estimated` stamped on +# the output — never silently. The provider `usage` numbers need no tokenizer and are exact. +# +# Usage: +# python3 bench/tokenaudit/loop_ledger.py ~/.claude/projects/ [--json OUT] [--sessions] + +import argparse +import collections +import glob +import json +import os +import re +import statistics +import sys + +NATIVE_TOOLS = frozenset(("Read", "Grep", "Glob", "NotebookRead")) +SWEEP_LOOKAHEAD = 3 # the substitution meter's post-call sweep definition + +# ── THE BASH CLASSIFIER IS THE METER'S, DELIBERATELY ────────────────────────────────────────────── +# The first version of this file matched a retrieval command only at the START of the line, and on +# this operator's log that under-counted native retrieval by ~3x: almost every Bash line here begins +# with plumbing (`cd X && grep …`, `echo "=== x ==="; sed -n …`), which is the exact gap +# hooks/ripwire-nudge.sh recorded and fixed in its 2026-08-12 classifier-gap round. The rules below +# are that function's, ported: walk the SEQUENCED segments (`;`, `&&`, `||`) and stop at the first +# the head table decides; do NOT walk pipeline stages (a `| grep` filters the FIRST command's output +# and is not a second observation); `grep -c`/`-q` is a poll, not a search; `cat > f` is a write; +# `sed -n` is a whole-file read; `ls -R` is a walk. The parity is the point — the two instruments +# measure the same population differently ONLY where they are meant to, so a disagreement between +# this file and bench/substitution_report.py is a finding rather than a definition mismatch. +RIPWIRE_WORD = re.compile(r"(^|[\s;&|(`\"'])(\./)?(build/|asan/)?ripwire(?![\w/])") +SEQ_SPLIT = re.compile(r"(?:;|&&|\|\||\n)") +GREP_POLL = re.compile(r"(^|\s)(-[A-Za-z]*[cq][A-Za-z]*|--count|--quiet)(\s|$)") +SED_QUIET = re.compile(r"(^|\s)-[A-Za-z]*n(\s|$)") +LS_WALK = re.compile(r"(^|\s)-[A-Za-z]*R[A-Za-z]*(\s|$)|--recursive(\s|$)") +AWK_PATTERN = re.compile(r"['\"]/") + + +def classify_segment(seg): + words = seg.strip().split() + if not words: + return None + lead = words[0] + if lead == "sudo" and len(words) > 1: + words = words[1:] + lead = words[0] + sub = words[1] if len(words) > 1 else "" + base = lead.rsplit("/", 1)[-1] + if base == "ripwire": + return "ripwire" + if base in ("grep", "egrep", "fgrep", "zgrep", "rg", "ag", "ack", "ack-grep", "ugrep"): + return "other" if GREP_POLL.search(seg) else "native" + if base in ("ps", "pgrep"): + return "other" + if base in ("find", "fd", "fdfind"): + return "native" + if base in ("cat", "head", "tail", "less", "more", "bat", "nl", "tac"): + return "other" if sub.startswith(">") else "native" + if base == "ls": + return "native" if LS_WALK.search(seg) else None + if base in ("awk", "gawk", "mawk"): + return "native" if AWK_PATTERN.search(seg) else None + if base == "sed": + return "native" if SED_QUIET.search(seg) else None + if base == "git": + return "other" + return None + + +def classify(name, inp): + if isinstance(name, str) and name.startswith("mcp__ripwire__"): + return "ripwire" + if name in NATIVE_TOOLS: + return "native" + if name == "Bash": + cmd = inp.get("command") or "" if isinstance(inp, dict) else "" + # ripwire anywhere on the line wins before the walk, exactly as the meter's head table puts + # `ripwire` first: `ripwire . --for=x | head -40` is one ripwire call, not a `head`. + if RIPWIRE_WORD.search(cmd): + return "ripwire" + for seg in SEQ_SPLIT.split(cmd): + verdict = classify_segment(seg) + if verdict is not None: + return verdict + return "other" + + +def result_text(entry): + """The bytes a tool result actually put into the context window, as text.""" + out = [] + tur = entry.get("toolUseResult") + if isinstance(tur, str): + out.append(tur) + elif tur is not None: + out.append(json.dumps(tur, ensure_ascii=False)) + else: + msg = entry.get("message") or {} + content = msg.get("content") + if isinstance(content, list): + for b in content: + if isinstance(b, dict) and b.get("type") == "tool_result": + c = b.get("content") + out.append(c if isinstance(c, str) else json.dumps(c, ensure_ascii=False)) + return "".join(out) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("project_dir") + ap.add_argument("--json", default=None) + ap.add_argument("--sessions", action="store_true", help="print per-session rows (local debugging only)") + args = ap.parse_args() + + try: + import tiktoken + enc = tiktoken.get_encoding("o200k_base") + def tok(s): + return len(enc.encode(s, disallowed_special=())) + tokenizer = "o200k_base" + except Exception: + def tok(s): + return int(len(s.encode("utf-8", "replace")) / 2.5 + 0.5) + tokenizer = "estimated(bytes/2.5)" + + files = sorted(glob.glob(os.path.join(os.path.expanduser(args.project_dir), "*.jsonl"))) + sessions = {} + + for path in files: + # calls: ordered [(family, tool_use_id)]; results: tool_use_id -> tokens + calls = [] + results = {} + usage = collections.Counter() + sid = None + with open(path, errors="replace") as f: + for line in f: + line = line.strip() + if not line: + continue + try: + e = json.loads(line) + except Exception: + continue + sid = sid or e.get("sessionId") + msg = e.get("message") or {} + if e.get("type") == "assistant": + u = msg.get("usage") or {} + for k in ("input_tokens", "output_tokens", + "cache_creation_input_tokens", "cache_read_input_tokens"): + v = u.get(k) + if isinstance(v, int): + usage[k] += v + content = msg.get("content") + if isinstance(content, list): + for b in content: + if isinstance(b, dict) and b.get("type") == "tool_use": + calls.append((classify(b.get("name"), b.get("input")), b.get("id"))) + elif e.get("type") == "user": + ids = [] + content = msg.get("content") + if isinstance(content, list): + for b in content: + if isinstance(b, dict) and b.get("type") == "tool_result": + ids.append(b.get("tool_use_id")) + if ids: + t = tok(result_text(e)) + # a result block carries one id in practice; split evenly if ever more + for i in ids: + results[i] = results.get(i, 0) + t // max(1, len(ids)) + if not calls: + continue + + fam_tokens = collections.Counter() + fam_calls = collections.Counter() + for fam, cid in calls: + fam_calls[fam] += 1 + fam_tokens[fam] += results.get(cid, 0) + + nonterminal = 0 + rip_total = 0 + for i, (fam, _) in enumerate(calls): + if fam != "ripwire": + continue + rip_total += 1 + if any(calls[j][0] == "native" for j in range(i + 1, min(i + 1 + SWEEP_LOOKAHEAD, len(calls)))): + nonterminal += 1 + + sessions[sid or os.path.basename(path)] = { + "calls": dict(fam_calls), + "result_tokens": dict(fam_tokens), + "ripwire_calls": rip_total, + "nonterminal_ripwire_calls": nonterminal, + "provider_usage": dict(usage), + } + + # ── aggregates only ─────────────────────────────────────────────────────────────────────────── + tot = collections.Counter() + tot_calls = collections.Counter() + usage_tot = collections.Counter() + rip = nonterm = 0 + with_rip = [] + for sid, s in sessions.items(): + for k, v in s["result_tokens"].items(): + tot[k] += v + for k, v in s["calls"].items(): + tot_calls[k] += v + for k, v in s["provider_usage"].items(): + usage_tot[k] += v + rip += s["ripwire_calls"] + nonterm += s["nonterminal_ripwire_calls"] + if s["ripwire_calls"]: + with_rip.append(s) + + report = { + "schema": "ripwire.loopledger/v1", + "tokenizer": tokenizer, + "sweep_lookahead": SWEEP_LOOKAHEAD, + "sessions": len(sessions), + "sessions_with_a_ripwire_call": len(with_rip), + "tool_calls": dict(tot_calls), + "tool_result_tokens": dict(tot), + "ripwire_calls": rip, + "nonterminal_ripwire_calls": nonterm, + "provider_usage_tokens": dict(usage_tot), + } + if rip: + report["nonterminality_rate"] = round(nonterm / rip, 4) + n = tot["native"] + tot["ripwire"] + if n: + report["ripwire_share_of_retrieval_tokens"] = round(tot["ripwire"] / n, 4) + if tot_calls["native"] + tot_calls["ripwire"]: + report["ripwire_share_of_retrieval_calls"] = round( + tot_calls["ripwire"] / (tot_calls["native"] + tot_calls["ripwire"]), 4) + for fam in ("ripwire", "native"): + per = [s["result_tokens"].get(fam, 0) / s["calls"][fam] + for s in sessions.values() if s["calls"].get(fam)] + if per: + report["median_tokens_per_%s_call" % fam] = round(statistics.median(per), 1) + report["mean_tokens_per_%s_call" % fam] = round(statistics.mean(per), 1) + if usage_tot: + billed = sum(usage_tot.values()) + if billed: + report["tool_result_share_of_billed_tokens"] = round( + (tot["ripwire"] + tot["native"] + tot["other"]) / billed, 4) + + json.dump(report, sys.stdout, indent=1, sort_keys=True) + sys.stdout.write("\n") + if args.json: + with open(args.json, "w") as f: + json.dump(report, f, indent=1, sort_keys=True) + f.write("\n") + if args.sessions: + for sid, s in sorted(sessions.items()): + sys.stderr.write("%s %s\n" % (sid[:8], json.dumps(s, sort_keys=True))) + + +if __name__ == "__main__": + main() diff --git a/bench/tokenaudit/pin.py b/bench/tokenaudit/pin.py new file mode 100644 index 00000000..09f2a03c --- /dev/null +++ b/bench/tokenaudit/pin.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +# pin.py — regenerate test/estcalib.manifest: the REAL o200k_base / cl100k_base token counts of +# ripwire's output on the frozen fixture test/estcalibfix, for the invocations that print est_tokens=. +# +# WHY A MANIFEST AND NOT A GATE THAT TOKENIZES. tiktoken is a Python package. G3 is one deterministic +# build step with nothing host-installed, and every gate in test/ runs on a machine that has bash, +# python3's STDLIB and the binary — nothing else. A gate that imported tiktoken would be a dependency +# the build contract forbids, and one that shelled out to a network-fetching encoder would be worse. +# So the tokenizer runs HERE, by hand, and writes numbers; the gate (test/tokenbudgetcheck.sh #18) +# reads those numbers and needs no package at all. Same shape as test/printf_parity.manifest. +# +# WHY THE PINS DO NOT ROT. The fixture is copied to a temp dir OUTSIDE any repository and crawled by a +# RELATIVE path, so the output carries no `at="+dirty"` stamp (nothing to stamp) and root="f" is +# one byte on every machine. The only thing that can move these numbers is the fixture (frozen) or the +# emitter (which is exactly what the gate is for). +# +# Usage: python3 bench/tokenaudit/pin.py --bin build/ripwire # needs tiktoken in the ambient python + +import argparse +import os +import shutil +import subprocess +import sys +import tempfile + +# The pinned invocations: label, then argv AFTER the root. No argument may contain a space — the +# manifest is space-separated and the gate splits on whitespace, exactly like printf_parity.manifest. +# Only verbs that PRINT est_tokens= belong here; a verb that prints none has nothing to calibrate +# (that absence is itself reported in bench/tokenaudit/README.md, not gated here). +PINNED = [ + ("map", []), + ("map-topk5", ["--top-k=5"]), + ("metrics", ["--metrics"]), + ("pack-signatures", ["--pack-signatures"]), + ("expand", ["--expand=billableTotal"]), + ("for-named", ["--for=billableTotal"]), + ("for-budgeted", ["--for=dedupe", "--token-budget=400"]), + ("pack-task", ["--pack-task=dedupe"]), +] + +EST = b'est_tokens="' + + +def est_of(out: bytes): + i = out.find(EST) + if i < 0: + return None + j = out.find(b'"', i + len(EST)) + return int(out[i + len(EST):j]) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--bin", default="build/ripwire") + ap.add_argument("--fixture", default="test/estcalibfix") + ap.add_argument("--out", default="test/estcalib.manifest") + args = ap.parse_args() + + import tiktoken + o200k = tiktoken.get_encoding("o200k_base") + cl100k = tiktoken.get_encoding("cl100k_base") + + binpath = os.path.abspath(args.bin) + fixture = os.path.abspath(args.fixture) + out_path = os.path.abspath(args.out) + + lines = [ + "# est_calib pins — REAL tokenizer counts of ripwire's output on test/estcalibfix.", + "# label o200k cl100k est_at_pin_time argv... (space-separated; no argv may contain a space)", + "# Regenerate: python3 bench/tokenaudit/pin.py --bin build/ripwire (needs tiktoken)", + "# Read by: test/tokenbudgetcheck.sh #18 — the calibration band. See bench/tokenaudit/README.md.", + ] + tmp = tempfile.mkdtemp() + try: + shutil.copytree(fixture, os.path.join(tmp, "f")) + for label, argv in PINNED: + p = subprocess.run([binpath, "f"] + argv, capture_output=True, cwd=tmp) + if p.returncode != 0: + print("pin.py: '%s' exited %d" % (label, p.returncode), file=sys.stderr) + return 1 + est = est_of(p.stdout) + if est is None: + print("pin.py: '%s' printed no est_tokens — remove it from PINNED" % label, file=sys.stderr) + return 1 + text = p.stdout.decode("utf-8", "replace") + lines.append("%s %d %d %d %s" % (label, len(o200k.encode(text)), + len(cl100k.encode(text)), est, " ".join(argv))) + finally: + shutil.rmtree(tmp, ignore_errors=True) + + with open(out_path, "w") as f: + f.write("\n".join(lines) + "\n") + print("wrote", out_path) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bench/tokenaudit/results/loopledger-2026-09-09.json b/bench/tokenaudit/results/loopledger-2026-09-09.json new file mode 100644 index 00000000..c1349908 --- /dev/null +++ b/bench/tokenaudit/results/loopledger-2026-09-09.json @@ -0,0 +1,33 @@ +{ + "mean_tokens_per_native_call": 5700.5, + "mean_tokens_per_ripwire_call": 406.7, + "median_tokens_per_native_call": 716.7, + "median_tokens_per_ripwire_call": 333.0, + "nonterminal_ripwire_calls": 1666, + "nonterminality_rate": 0.4821, + "provider_usage_tokens": { + "cache_creation_input_tokens": 221210191, + "cache_read_input_tokens": 12846299888, + "input_tokens": 286905, + "output_tokens": 43186745 + }, + "ripwire_calls": 3456, + "ripwire_share_of_retrieval_calls": 0.422, + "ripwire_share_of_retrieval_tokens": 0.0426, + "schema": "ripwire.loopledger/v1", + "sessions": 78, + "sessions_with_a_ripwire_call": 67, + "sweep_lookahead": 3, + "tokenizer": "o200k_base", + "tool_calls": { + "native": 4733, + "other": 8921, + "ripwire": 3456 + }, + "tool_result_share_of_billed_tokens": 0.0025, + "tool_result_tokens": { + "native": 26083509, + "other": 5095267, + "ripwire": 1159938 + } +} diff --git a/bench/tokenaudit/results/tokenaudit-2026-09-09.json b/bench/tokenaudit/results/tokenaudit-2026-09-09.json new file mode 100644 index 00000000..d1d5ecd2 --- /dev/null +++ b/bench/tokenaudit/results/tokenaudit-2026-09-09.json @@ -0,0 +1,906 @@ +{ + "bin": "/Users/qgames/AppDevelopLocal/project2/rw-hv-B/build/ripwire", + "corpora": { + "canyonraid48": { + "file": "math/doctest.h", + "root": "/Users/qgames/AppDevelopLocal/project2/canyonraid48", + "rows": [ + { + "argv": [], + "bytes": 29123, + "bytes_per_o200k": 2.7159, + "cl100k": 10570, + "compact_bytes": 27881, + "compact_err_pct": 13.85, + "compact_est_tokens": 11773, + "compact_o200k": 10341, + "err_pct": 9.79, + "est_tokens": 11773, + "legend_bytes": 1242, + "legend_o200k": 382, + "legend_share_pct": 3.56, + "o200k": 10723, + "rc": 0, + "verb": "map" + }, + { + "argv": [ + "--top-k=10" + ], + "bytes": 2936, + "bytes_per_o200k": 2.9478, + "cl100k": 996, + "compact_bytes": 1694, + "compact_err_pct": 93.0, + "compact_est_tokens": 1185, + "compact_o200k": 614, + "err_pct": 18.98, + "est_tokens": 1185, + "legend_bytes": 1242, + "legend_o200k": 382, + "legend_share_pct": 38.35, + "o200k": 996, + "rc": 0, + "verb": "map-topk10" + }, + { + "argv": [ + "--json" + ], + "bytes": 29377, + "bytes_per_o200k": 2.9545, + "cl100k": 9942, + "est_tokens": null, + "o200k": 9943, + "rc": 0, + "verb": "map-json" + }, + { + "argv": [ + "--metrics", + "--top-k=40" + ], + "bytes": 14746, + "bytes_per_o200k": 2.6512, + "cl100k": 5471, + "compact_bytes": 12103, + "compact_err_pct": 24.89, + "compact_est_tokens": 5961, + "compact_o200k": 4773, + "err_pct": 7.17, + "est_tokens": 5961, + "legend_bytes": 2643, + "legend_o200k": 789, + "legend_share_pct": 14.19, + "o200k": 5562, + "rc": 0, + "verb": "metrics" + }, + { + "argv": [ + "--pack-signatures", + "--top-k=40" + ], + "bytes": 17065, + "bytes_per_o200k": 2.9478, + "cl100k": 5703, + "compact_bytes": 15790, + "compact_err_pct": 27.27, + "compact_est_tokens": 6860, + "compact_o200k": 5390, + "err_pct": 18.5, + "est_tokens": 6860, + "legend_bytes": 1275, + "legend_o200k": 399, + "legend_share_pct": 6.89, + "o200k": 5789, + "rc": 0, + "verb": "pack-signatures" + }, + { + "argv": [ + "--for=how are tokens counted and budgeted" + ], + "bytes": 9021, + "bytes_per_o200k": 3.1366, + "cl100k": 2904, + "compact_bytes": 8963, + "compact_err_pct": 22.35, + "compact_est_tokens": 3585, + "compact_o200k": 2930, + "err_pct": 25.45, + "est_tokens": 3608, + "legend_bytes": 58, + "legend_o200k": -54, + "legend_share_pct": -1.88, + "o200k": 2876, + "rc": 0, + "verb": "for-conceptual" + }, + { + "argv": [ + "--for=TEST_CASE" + ], + "bytes": 3318, + "bytes_per_o200k": 3.5186, + "cl100k": 935, + "compact_bytes": 2613, + "compact_err_pct": 26.36, + "compact_est_tokens": 997, + "compact_o200k": 789, + "err_pct": 35.63, + "est_tokens": 1279, + "legend_bytes": 705, + "legend_o200k": 154, + "legend_share_pct": 16.33, + "o200k": 943, + "rc": 0, + "verb": "for-named" + }, + { + "argv": [ + "--for=how are tokens counted and budgeted", + "--json" + ], + "bytes": 8274, + "bytes_per_o200k": 3.1, + "cl100k": 2747, + "est_tokens": null, + "o200k": 2669, + "rc": 0, + "verb": "for-json" + }, + { + "argv": [ + "--pack-task=fix the token estimate" + ], + "bytes": 11449, + "bytes_per_o200k": 3.3793, + "cl100k": 3423, + "compact_bytes": 10092, + "compact_err_pct": 38.76, + "compact_est_tokens": 4221, + "compact_o200k": 3042, + "err_pct": 24.59, + "est_tokens": 4221, + "legend_bytes": 1357, + "legend_o200k": 346, + "legend_share_pct": 10.21, + "o200k": 3388, + "rc": 0, + "verb": "pack-task" + }, + { + "argv": [ + "--expand=TEST_CASE" + ], + "bytes": 1695, + "bytes_per_o200k": 3.4173, + "cl100k": 496, + "compact_bytes": 1229, + "compact_err_pct": 2.27, + "compact_est_tokens": 405, + "compact_o200k": 396, + "err_pct": -18.35, + "est_tokens": 405, + "legend_bytes": 466, + "legend_o200k": 100, + "legend_share_pct": 20.16, + "o200k": 496, + "rc": 0, + "verb": "expand" + }, + { + "argv": [ + "--callers=TEST_CASE" + ], + "bytes": 8264, + "bytes_per_o200k": 3.7292, + "cl100k": 2182, + "compact_bytes": 4956, + "compact_est_tokens": null, + "compact_o200k": 1478, + "est_tokens": null, + "legend_bytes": 3308, + "legend_o200k": 738, + "legend_share_pct": 33.3, + "o200k": 2216, + "rc": 0, + "verb": "callers" + }, + { + "argv": [ + "--callees=TEST_CASE" + ], + "bytes": 3838, + "bytes_per_o200k": 4.1627, + "cl100k": 916, + "compact_bytes": 659, + "compact_est_tokens": null, + "compact_o200k": 209, + "est_tokens": null, + "legend_bytes": 3179, + "legend_o200k": 713, + "legend_share_pct": 77.33, + "o200k": 922, + "rc": 0, + "verb": "callees" + }, + { + "argv": [ + "--impact=TEST_CASE" + ], + "bytes": 10984, + "bytes_per_o200k": 3.5512, + "cl100k": 3063, + "compact_bytes": 7594, + "compact_est_tokens": null, + "compact_o200k": 2321, + "est_tokens": null, + "legend_bytes": 3390, + "legend_o200k": 772, + "legend_share_pct": 24.96, + "o200k": 3093, + "rc": 0, + "verb": "impact" + }, + { + "argv": [ + "--uses=TEST_CASE" + ], + "bytes": 10235, + "bytes_per_o200k": 3.5464, + "cl100k": 2785, + "compact_bytes": 6246, + "compact_est_tokens": null, + "compact_o200k": 1994, + "est_tokens": null, + "legend_bytes": 3989, + "legend_o200k": 892, + "legend_share_pct": 30.91, + "o200k": 2886, + "rc": 0, + "verb": "uses" + }, + { + "argv": [ + "--around=TEST_CASE" + ], + "bytes": 9035, + "bytes_per_o200k": 2.819, + "cl100k": 3047, + "compact_bytes": 7605, + "compact_err_pct": 30.95, + "compact_est_tokens": 3643, + "compact_o200k": 2782, + "err_pct": 13.67, + "est_tokens": 3643, + "legend_bytes": 1430, + "legend_o200k": 423, + "legend_share_pct": 13.2, + "o200k": 3205, + "rc": 0, + "verb": "around" + }, + { + "argv": [ + "--edit-check=TEST_CASE" + ], + "bytes": 348224, + "bytes_per_o200k": 3.5172, + "cl100k": 97575, + "compact_bytes": 342339, + "compact_est_tokens": null, + "compact_o200k": 97709, + "est_tokens": null, + "legend_bytes": 5885, + "legend_o200k": 1297, + "legend_share_pct": 1.31, + "o200k": 99006, + "rc": 0, + "verb": "edit-check" + }, + { + "argv": [ + "--grep=token" + ], + "bytes": 32149, + "bytes_per_o200k": 3.5809, + "cl100k": 8890, + "compact_bytes": 25881, + "compact_est_tokens": null, + "compact_o200k": 7564, + "est_tokens": null, + "legend_bytes": 6268, + "legend_o200k": 1414, + "legend_share_pct": 15.75, + "o200k": 8978, + "rc": 0, + "verb": "grep" + }, + { + "argv": [ + "--hotspots" + ], + "bytes": 6582, + "bytes_per_o200k": 2.9555, + "cl100k": 2222, + "compact_bytes": 5465, + "compact_est_tokens": null, + "compact_o200k": 1987, + "est_tokens": null, + "legend_bytes": 1117, + "legend_o200k": 240, + "legend_share_pct": 10.78, + "o200k": 2227, + "rc": 0, + "verb": "hotspots" + }, + { + "argv": [ + "--lint" + ], + "bytes": 56589, + "bytes_per_o200k": 3.0643, + "cl100k": 18607, + "compact_bytes": 53998, + "compact_est_tokens": null, + "compact_o200k": 17863, + "est_tokens": null, + "legend_bytes": 2591, + "legend_o200k": 604, + "legend_share_pct": 3.27, + "o200k": 18467, + "rc": 0, + "verb": "lint" + }, + { + "argv": [ + "--tree" + ], + "bytes": 13265, + "bytes_per_o200k": 2.9478, + "cl100k": 4369, + "compact_bytes": 12543, + "compact_est_tokens": null, + "compact_o200k": 4344, + "est_tokens": null, + "legend_bytes": 722, + "legend_o200k": 156, + "legend_share_pct": 3.47, + "o200k": 4500, + "rc": 0, + "verb": "tree" + }, + { + "argv": [ + "--clones" + ], + "bytes": 19145, + "bytes_per_o200k": 3.1432, + "cl100k": 6093, + "compact_bytes": 16115, + "compact_est_tokens": null, + "compact_o200k": 5424, + "est_tokens": null, + "legend_bytes": 3030, + "legend_o200k": 667, + "legend_share_pct": 10.95, + "o200k": 6091, + "rc": 0, + "verb": "clones" + }, + { + "argv": [ + "--situ" + ], + "bytes": 1826, + "bytes_per_o200k": 3.7572, + "cl100k": 486, + "est_tokens": null, + "o200k": 486, + "rc": 0, + "verb": "situ" + }, + { + "argv": [ + "--recall=how are tokens counted" + ], + "bytes": 11912, + "bytes_per_o200k": 3.6185, + "cl100k": 3328, + "est_tokens": null, + "o200k": 3292, + "rc": 0, + "verb": "recall" + }, + { + "argv": [ + "--test-gate" + ], + "bytes": 3232, + "bytes_per_o200k": 3.843, + "cl100k": 833, + "compact_refused": "", + "est_tokens": null, + "o200k": 841, + "rc": 4, + "verb": "test-gate" + }, + { + "argv": [ + "--affected=math/doctest.h" + ], + "bytes": 17006, + "bytes_per_o200k": 3.2849, + "cl100k": 5081, + "compact_bytes": 15537, + "compact_est_tokens": null, + "compact_o200k": 4844, + "est_tokens": null, + "legend_bytes": 1469, + "legend_o200k": 333, + "legend_share_pct": 6.43, + "o200k": 5177, + "rc": 0, + "verb": "affected" + } + ], + "symbol": "TEST_CASE" + }, + "self": { + "file": "src/infra/svector.h", + "root": "/Users/qgames/AppDevelopLocal/project2/rw-hv-B", + "rows": [ + { + "argv": [], + "bytes": 24756, + "bytes_per_o200k": 2.5388, + "cl100k": 9561, + "compact_bytes": 23762, + "compact_err_pct": 6.15, + "compact_est_tokens": 10022, + "compact_o200k": 9441, + "err_pct": 2.78, + "est_tokens": 10022, + "legend_bytes": 994, + "legend_o200k": 310, + "legend_share_pct": 3.18, + "o200k": 9751, + "rc": 0, + "verb": "map" + }, + { + "argv": [ + "--top-k=10" + ], + "bytes": 2860, + "bytes_per_o200k": 2.8686, + "cl100k": 985, + "compact_bytes": 1866, + "compact_err_pct": 68.12, + "compact_est_tokens": 1155, + "compact_o200k": 687, + "err_pct": 15.85, + "est_tokens": 1155, + "legend_bytes": 994, + "legend_o200k": 310, + "legend_share_pct": 31.09, + "o200k": 997, + "rc": 0, + "verb": "map-topk10" + }, + { + "argv": [ + "--json" + ], + "bytes": 25352, + "bytes_per_o200k": 2.7521, + "cl100k": 9091, + "est_tokens": null, + "o200k": 9212, + "rc": 0, + "verb": "map-json" + }, + { + "argv": [ + "--metrics", + "--top-k=40" + ], + "bytes": 12016, + "bytes_per_o200k": 2.5824, + "cl100k": 4623, + "compact_bytes": 9621, + "compact_err_pct": 23.45, + "compact_est_tokens": 4859, + "compact_o200k": 3936, + "err_pct": 4.43, + "est_tokens": 4859, + "legend_bytes": 2395, + "legend_o200k": 717, + "legend_share_pct": 15.41, + "o200k": 4653, + "rc": 0, + "verb": "metrics" + }, + { + "argv": [ + "--pack-signatures", + "--top-k=40" + ], + "bytes": 15710, + "bytes_per_o200k": 2.913, + "cl100k": 5353, + "compact_bytes": 14683, + "compact_err_pct": 24.5, + "compact_est_tokens": 6307, + "compact_o200k": 5066, + "err_pct": 16.95, + "est_tokens": 6307, + "legend_bytes": 1027, + "legend_o200k": 327, + "legend_share_pct": 6.06, + "o200k": 5393, + "rc": 0, + "verb": "pack-signatures" + }, + { + "argv": [ + "--for=how are tokens counted and budgeted" + ], + "bytes": 9725, + "bytes_per_o200k": 3.0486, + "cl100k": 3160, + "compact_bytes": 9562, + "compact_err_pct": 18.94, + "compact_est_tokens": 3825, + "compact_o200k": 3216, + "err_pct": 21.94, + "est_tokens": 3890, + "legend_bytes": 163, + "legend_o200k": -26, + "legend_share_pct": -0.82, + "o200k": 3190, + "rc": 0, + "verb": "for-conceptual" + }, + { + "argv": [ + "--for=buf" + ], + "bytes": 3822, + "bytes_per_o200k": 3.2472, + "cl100k": 1173, + "compact_bytes": 3124, + "compact_err_pct": 17.56, + "compact_est_tokens": 1205, + "compact_o200k": 1025, + "err_pct": 26.08, + "est_tokens": 1484, + "legend_bytes": 698, + "legend_o200k": 152, + "legend_share_pct": 12.91, + "o200k": 1177, + "rc": 0, + "verb": "for-named" + }, + { + "argv": [ + "--for=how are tokens counted and budgeted", + "--json" + ], + "bytes": 7774, + "bytes_per_o200k": 3.051, + "cl100k": 2560, + "est_tokens": null, + "o200k": 2548, + "rc": 0, + "verb": "for-json" + }, + { + "argv": [ + "--pack-task=fix the token estimate" + ], + "bytes": 11671, + "bytes_per_o200k": 3.3673, + "cl100k": 3456, + "compact_bytes": 10410, + "compact_err_pct": 31.04, + "compact_est_tokens": 4120, + "compact_o200k": 3144, + "err_pct": 18.87, + "est_tokens": 4120, + "legend_bytes": 1261, + "legend_o200k": 322, + "legend_share_pct": 9.29, + "o200k": 3466, + "rc": 0, + "verb": "pack-task" + }, + { + "argv": [ + "--expand=buf" + ], + "bytes": 26539, + "bytes_per_o200k": 2.5814, + "cl100k": 10091, + "compact_bytes": 24634, + "compact_err_pct": 7.54, + "compact_est_tokens": 10474, + "compact_o200k": 9740, + "err_pct": 1.88, + "est_tokens": 10474, + "legend_bytes": 1905, + "legend_o200k": 541, + "legend_share_pct": 5.26, + "o200k": 10281, + "rc": 0, + "verb": "expand" + }, + { + "argv": [ + "--callers=buf" + ], + "bytes": 5373, + "bytes_per_o200k": 3.4487, + "cl100k": 1521, + "compact_bytes": 2396, + "compact_est_tokens": null, + "compact_o200k": 889, + "est_tokens": null, + "legend_bytes": 2977, + "legend_o200k": 669, + "legend_share_pct": 42.94, + "o200k": 1558, + "rc": 0, + "verb": "callers" + }, + { + "argv": [ + "--callees=buf" + ], + "bytes": 3750, + "bytes_per_o200k": 4.1993, + "cl100k": 887, + "compact_bytes": 571, + "compact_est_tokens": null, + "compact_o200k": 180, + "est_tokens": null, + "legend_bytes": 3179, + "legend_o200k": 713, + "legend_share_pct": 79.84, + "o200k": 893, + "rc": 0, + "verb": "callees" + }, + { + "argv": [ + "--impact=buf" + ], + "bytes": 6674, + "bytes_per_o200k": 3.3023, + "cl100k": 1969, + "compact_bytes": 3284, + "compact_est_tokens": null, + "compact_o200k": 1249, + "est_tokens": null, + "legend_bytes": 3390, + "legend_o200k": 772, + "legend_share_pct": 38.2, + "o200k": 2021, + "rc": 0, + "verb": "impact" + }, + { + "argv": [ + "--uses=buf" + ], + "bytes": 7917, + "bytes_per_o200k": 3.3964, + "cl100k": 2285, + "compact_bytes": 4259, + "compact_est_tokens": null, + "compact_o200k": 1508, + "est_tokens": null, + "legend_bytes": 3658, + "legend_o200k": 823, + "legend_share_pct": 35.31, + "o200k": 2331, + "rc": 0, + "verb": "uses" + }, + { + "argv": [ + "--around=buf" + ], + "bytes": 6033, + "bytes_per_o200k": 2.8404, + "cl100k": 2078, + "compact_bytes": 4851, + "compact_err_pct": 37.39, + "compact_est_tokens": 2436, + "compact_o200k": 1773, + "err_pct": 14.69, + "est_tokens": 2436, + "legend_bytes": 1182, + "legend_o200k": 351, + "legend_share_pct": 16.53, + "o200k": 2124, + "rc": 0, + "verb": "around" + }, + { + "argv": [ + "--edit-check=buf" + ], + "bytes": 7986, + "bytes_per_o200k": 3.8748, + "cl100k": 2013, + "compact_bytes": 2101, + "compact_est_tokens": null, + "compact_o200k": 764, + "est_tokens": null, + "legend_bytes": 5885, + "legend_o200k": 1297, + "legend_share_pct": 62.93, + "o200k": 2061, + "rc": 0, + "verb": "edit-check" + }, + { + "argv": [ + "--grep=token" + ], + "bytes": 68695, + "bytes_per_o200k": 3.1402, + "cl100k": 21522, + "compact_bytes": 62482, + "compact_est_tokens": null, + "compact_o200k": 20477, + "est_tokens": null, + "legend_bytes": 6213, + "legend_o200k": 1399, + "legend_share_pct": 6.4, + "o200k": 21876, + "rc": 0, + "verb": "grep" + }, + { + "argv": [ + "--hotspots" + ], + "bytes": 6002, + "bytes_per_o200k": 2.8311, + "cl100k": 2077, + "compact_bytes": 4885, + "compact_est_tokens": null, + "compact_o200k": 1880, + "est_tokens": null, + "legend_bytes": 1117, + "legend_o200k": 240, + "legend_share_pct": 11.32, + "o200k": 2120, + "rc": 0, + "verb": "hotspots" + }, + { + "argv": [ + "--lint" + ], + "bytes": 56147, + "bytes_per_o200k": 2.9386, + "cl100k": 18957, + "compact_bytes": 53556, + "compact_est_tokens": null, + "compact_o200k": 18503, + "est_tokens": null, + "legend_bytes": 2591, + "legend_o200k": 604, + "legend_share_pct": 3.16, + "o200k": 19107, + "rc": 0, + "verb": "lint" + }, + { + "argv": [ + "--tree" + ], + "bytes": 11755, + "bytes_per_o200k": 2.7574, + "cl100k": 4092, + "compact_bytes": 11033, + "compact_est_tokens": null, + "compact_o200k": 4107, + "est_tokens": null, + "legend_bytes": 722, + "legend_o200k": 156, + "legend_share_pct": 3.66, + "o200k": 4263, + "rc": 0, + "verb": "tree" + }, + { + "argv": [ + "--clones" + ], + "bytes": 18652, + "bytes_per_o200k": 2.9819, + "cl100k": 6128, + "compact_bytes": 15622, + "compact_est_tokens": null, + "compact_o200k": 5588, + "est_tokens": null, + "legend_bytes": 3030, + "legend_o200k": 667, + "legend_share_pct": 10.66, + "o200k": 6255, + "rc": 0, + "verb": "clones" + }, + { + "argv": [ + "--situ" + ], + "bytes": 210, + "bytes_per_o200k": 3.3333, + "cl100k": 64, + "est_tokens": null, + "o200k": 63, + "rc": 0, + "verb": "situ" + }, + { + "argv": [ + "--recall=how are tokens counted" + ], + "bytes": 15246, + "bytes_per_o200k": 3.7672, + "cl100k": 4050, + "est_tokens": null, + "o200k": 4047, + "rc": 0, + "verb": "recall" + }, + { + "argv": [ + "--test-gate" + ], + "bytes": 2078, + "bytes_per_o200k": 3.9134, + "cl100k": 526, + "compact_bytes": 805, + "compact_est_tokens": null, + "compact_o200k": 253, + "est_tokens": null, + "legend_bytes": 1273, + "legend_o200k": 278, + "legend_share_pct": 52.35, + "o200k": 531, + "rc": 0, + "verb": "test-gate" + }, + { + "argv": [ + "--affected=src/infra/svector.h" + ], + "bytes": 4267, + "bytes_per_o200k": 3.6039, + "cl100k": 1156, + "compact_bytes": 2798, + "compact_est_tokens": null, + "compact_o200k": 851, + "est_tokens": null, + "legend_bytes": 1469, + "legend_o200k": 333, + "legend_share_pct": 28.12, + "o200k": 1184, + "rc": 0, + "verb": "affected" + } + ], + "symbol": "buf" + } + }, + "schema": "ripwire.tokenaudit/v1", + "version": "ripwire 0.5.0 (dev, AppleClang 21.0.0.21000101, emit=std::print, built_from=4c10be9d7)" +} diff --git a/bench/tokenaudit/sweep.py b/bench/tokenaudit/sweep.py new file mode 100644 index 00000000..c3c10b0d --- /dev/null +++ b/bench/tokenaudit/sweep.py @@ -0,0 +1,195 @@ +#!/usr/bin/env python3 +# sweep.py — does ripwire's est_tokens= agree with a real tokenizer, per verb and per corpus? +# +# WHY THIS EXISTS. est_tokens is the number every --token-budget gates on and the number the owner +# named as the user-facing unit, and until this script it was validated only for its PROPERTIES +# (deterministic, present, monotone under a tighter budget — test/tokenbudgetcheck.sh) and by a +# MAPE the T1 author reported by hand in a write-up. Nothing in the tree re-derived the error against +# a tokenizer, and nothing measured it PER VERB. A single-corpus MAPE cannot see a per-verb bias: +# the divisor is a per-language content-byte model, and a --callers answer is ~90% markup while a +# --expand answer is ~70% body, so one rate serving both is a hypothesis, not a measurement. +# +# It also prices THE LEGEND. A third-party evaluation (callstack/agent-device #2400, 2026-09-08) +# measured ripwire spending ~10% MORE tokens than grep-and-read and traced it to the fixed per-call +# preamble. test/legendcostcheck.sh holds --help's BYTE claim to what the binary delivers; this +# script measures the same thing in the unit the claim is read in — real tokenizer tokens — by +# running each invocation twice, --legend=full and --legend=compact, on the same corpus. +# +# WHAT IS MEASURED, PER (corpus, invocation): +# bytes len(stdout) in bytes +# est_tokens the number ripwire printed, or null if the verb prints none <-- a finding in itself +# o200k / cl100k real token counts (tiktoken), the ground truth this file compares against +# err_pct 100 * (est - o200k) / o200k signed: negative = ripwire UNDER-reports its price +# legend_* the same numbers for --legend=compact, and the token delta = the legend's price +# +# WHAT IT IS NOT. Not a gate. tiktoken is not a build dependency and never will be (G3: one +# deterministic build step, nothing host-installed), and o200k_base is not Claude's tokenizer — +# it is the closest public stand-in, which is exactly why kTokenCalib was calibrated against it and +# why a gate that needs it at runtime would be a dependency problem. The gate this round adds +# (test/estcalibcheck.sh) reads PINNED counts out of a manifest this script writes; re-running this +# script is how the manifest is regenerated, deliberately by hand. +# +# Usage: +# python3 bench/tokenaudit/sweep.py --bin build/ripwire --out bench/tokenaudit/results/x.json \ +# [--corpus NAME=PATH ...] [--python PYTHON_WITH_TIKTOKEN] +# +# Determinism: the invocation table below is fixed and ordered; symbol arguments are DERIVED from +# each corpus's own map (highest-ranked symbol with callers) so the table transfers to a corpus this +# file has never seen, and the derived symbol is recorded in the results so a rerun is auditable. + +import argparse +import json +import os +import re +import subprocess +import sys + +EST_RE = re.compile(rb'est_tokens="(\d+)"') +SYM_RE = re.compile(rb']*\bn="([^"]+)"') +FILE_RE = re.compile(rb' row of the map (rank 1) and the first file. Deterministic + because the map is; recorded in the output because a number is unauditable without its input. + """ + out, _, _ = run(binpath, root, ["--top-k=40"]) + syms = SYM_RE.findall(out) + files = FILE_RE.findall(out) + sym = syms[0].decode() if syms else "main" + fil = files[0].decode() if files else "" + # Prefer a symbol that actually has callers AND is unambiguous, else --callers/--impact answer + # nothing and --edit-check refuses — the legend share would then be measured on an empty or + # refused document (true, but not the case anyone runs). The FIRST candidate that satisfies both + # wins, so the choice stays deterministic and is recorded in the results. + for cand in syms[:20]: + c = cand.decode() + co, _, rc = run(binpath, root, ["--callers=" + c, "--legend=compact"]) + if rc != 0 or co.count(b" 0.0 ); diff --git a/test/estcalib.manifest b/test/estcalib.manifest new file mode 100644 index 00000000..f877c433 --- /dev/null +++ b/test/estcalib.manifest @@ -0,0 +1,12 @@ +# est_calib pins — REAL tokenizer counts of ripwire's output on test/estcalibfix. +# label o200k cl100k est_at_pin_time argv... (space-separated; no argv may contain a space) +# Regenerate: python3 bench/tokenaudit/pin.py --bin build/ripwire (needs tiktoken) +# Read by: test/tokenbudgetcheck.sh #18 — the calibration band. See bench/tokenaudit/README.md. +map 1009 999 1123 +map-topk5 538 535 653 --top-k=5 +metrics 2381 2370 2509 --metrics +pack-signatures 1801 1784 2116 --pack-signatures +expand 543 544 454 --expand=billableTotal +for-named 789 785 1050 --for=billableTotal +for-budgeted 374 370 525 --for=dedupe --token-budget=400 +pack-task 882 862 1104 --pack-task=dedupe diff --git a/test/estcalibfix/ledger.cpp b/test/estcalibfix/ledger.cpp new file mode 100644 index 00000000..842a87e0 --- /dev/null +++ b/test/estcalibfix/ledger.cpp @@ -0,0 +1,86 @@ +// estcalibfix — a FROZEN corpus for the est_tokens calibration band (tokenbudgetcheck #18). +// Do not edit: test/estcalib.manifest pins real o200k_base token counts of ripwire's output on +// THIS tree, and any edit here invalidates every pin. Regenerate with bench/tokenaudit/pin.py. +#include +#include +#include + +namespace ledger +{ + +struct Entry +{ + std::string name; + std::size_t inputTokens; + std::size_t outputTokens; + std::size_t cacheReadTokens; +}; + +// Sum one class across the ledger. A zero here means "none found in these rows", never "none exists". +inline std::size_t sumInput( const std::vector& rows ) +{ + std::size_t total = 0; + for( const Entry& e : rows ) + { + total += e.inputTokens; + } + return total; +} + +inline std::size_t sumOutput( const std::vector& rows ) +{ + std::size_t total = 0; + for( const Entry& e : rows ) + { + total += e.outputTokens; + } + return total; +} + +// Cache reads are charged at a different rate than fresh input, so they are never folded into +// sumInput; a caller that wants one number asks for it and says which classes it merged. +inline std::size_t sumCacheRead( const std::vector& rows ) +{ + std::size_t total = 0; + for( const Entry& e : rows ) + { + total += e.cacheReadTokens; + } + return total; +} + +inline std::size_t billableTotal( const std::vector& rows ) +{ + return sumInput( rows ) + sumOutput( rows ) + sumCacheRead( rows ); +} + +inline bool isEmpty( const std::vector& rows ) +{ + return rows.empty(); +} + +class Report +{ +public: + explicit Report( std::vector rows ) : rows_( std::move( rows ) ) {} + + std::size_t total() const + { + return billableTotal( rows_ ); + } + + std::size_t rowCount() const + { + return rows_.size(); + } + + bool degenerate() const + { + return isEmpty( rows_ ) || total() == 0; + } + +private: + std::vector rows_; +}; + +} // namespace ledger diff --git a/test/estcalibfix/rates.py b/test/estcalibfix/rates.py new file mode 100644 index 00000000..cf793aec --- /dev/null +++ b/test/estcalibfix/rates.py @@ -0,0 +1,40 @@ +"""estcalibfix — FROZEN. See ledger.cpp's header; edits invalidate test/estcalib.manifest.""" + +DEFAULT_RATE = 2.5 +BODY_RATE = 3.8 + + +def bytes_per_token(language): + """Return the measured bytes-per-token rate for a language, or the mid-band default.""" + table = {"cpp": 2.46, "python": 2.36, "typescript": 2.59} + return table.get(language, DEFAULT_RATE) + + +def estimate_tokens(byte_count, language): + """Convert emitted bytes to a token estimate at the language's own rate.""" + rate = bytes_per_token(language) + if rate <= 0: + raise ValueError("a non-positive rate is a corrupt caller, never a runtime condition") + return int(byte_count / rate + 0.5) + + +def estimate_body_tokens(byte_count): + """Body text tokenizes leaner than signature markup; charge it at its own rate.""" + return int(byte_count / BODY_RATE + 0.5) + + +def signed_error_pct(estimate, real): + """Positive means the estimate over-reads the real count.""" + if real == 0: + return None + return 100.0 * (estimate - real) / real + + +def summarize(rows): + """rows: [(estimate, real)]. Returns (n, worst_over, worst_under) with None for an empty set.""" + if not rows: + return (0, None, None) + errs = [signed_error_pct(e, r) for e, r in rows if signed_error_pct(e, r) is not None] + if not errs: + return (len(rows), None, None) + return (len(rows), max(errs), min(errs)) diff --git a/test/estcalibfix/session.ts b/test/estcalibfix/session.ts new file mode 100644 index 00000000..a3187931 --- /dev/null +++ b/test/estcalibfix/session.ts @@ -0,0 +1,46 @@ +// estcalibfix — FROZEN. See ledger.cpp's header; edits invalidate test/estcalib.manifest. + +export interface UsageRecord { + sessionId: string; + model: string; + inputTokens: number; + outputTokens: number; + cacheReadTokens: number; + cacheWriteTokens: number; +} + +export function totalForRecord(record: UsageRecord): number { + return ( + record.inputTokens + + record.outputTokens + + record.cacheReadTokens + + record.cacheWriteTokens + ); +} + +export function groupBySession(records: UsageRecord[]): Map { + const out = new Map(); + for (const record of records) { + const prior = out.get(record.sessionId) ?? 0; + out.set(record.sessionId, prior + totalForRecord(record)); + } + return out; +} + +export function dedupe(records: UsageRecord[], keyOf: (r: UsageRecord) => string): UsageRecord[] { + const seen = new Set(); + const kept: UsageRecord[] = []; + for (const record of records) { + const key = keyOf(record); + if (seen.has(key)) { + continue; + } + seen.add(key); + kept.push(record); + } + return kept; +} + +export function isMeasured(record: UsageRecord): boolean { + return record.inputTokens > 0 || record.outputTokens > 0; +} diff --git a/test/tokenbudgetcheck.sh b/test/tokenbudgetcheck.sh index cd5ebd40..e3e0e5b4 100755 --- a/test/tokenbudgetcheck.sh +++ b/test/tokenbudgetcheck.sh @@ -318,5 +318,86 @@ else done fi +# ── #18: THE CALIBRATION BAND — est_tokens against a REAL tokenizer, pinned ──────────────────────────── +# THE GAP THIS CLOSES. Everything above measures the estimate's PROPERTIES: present, positive, +# deterministic, monotone under a tighter budget, and bounded by an allowance derived from the estimate's +# OWN constants. Not one arm asks whether the number is TRUE. This file's own header said so — "the +# MAPE-vs-tiktoken number is REPORTED by the agent in the T1 write-up (tiktoken isn't a build +# dependency)" — which makes the accuracy of the tool's most-quoted number a thing a human typed into a +# document once, in 2026-07, and nothing has re-derived since. METHODOLOGY §9 principle 6: measuring gets +# its own instrument. This is it. +# +# HOW IT AVOIDS THE DEPENDENCY. The tokenizer runs OUT OF BAND (bench/tokenaudit/pin.py, by hand, with +# tiktoken) and writes real o200k_base/cl100k_base counts into test/estcalib.manifest. This arm reads +# numbers. No package, no network, bash + python3's stdlib + the binary — the same contract as +# test/printf_parity.manifest. +# +# WHY THE PINS DO NOT ROT, and why the fixture is copied. test/estcalibfix is frozen, and the copy is +# crawled from a temp dir by a RELATIVE path: outside any repository there is no `at="+dirty"` stamp +# to change under a commit (or under a concurrent gate — the same shared-resource class this file's header +# records at #14), and root="f" is one byte on every machine, so a deep checkout path cannot move the +# count. Both artifacts that CAN move these numbers are the ones under test: the fixture and the emitter. +# +# THE BAND IS AN ENVELOPE, NOT A TARGET, and it is deliberately signed-asymmetric-free. Measured on three +# corpora (this fixture, this repository, a 1500-file private C++ tree; bench/tokenaudit/README.md) the +# signed error runs -18.4% to +41.7%: est_tokens OVER-reads the small legend-heavy bundles (the legend is +# English prose at ~4.4 B/tok charged at a ~2.5 B/tok signature rate) and UNDER-reads --expand's dense +# small bodies. So the band is +-, not a one-sided "never under-reads" floor: that claim was in +# src/serialize.h and this instrument is what disproved it. What the band DOES buy is that the error +# cannot silently double — a rate table edit, a new emitter charging bytes at the wrong rate, or a legend +# that grows into a document whose price is quoted at the markup rate all leave the band. +EST_MAN="$ROOT/test/estcalib.manifest" +EST_FIX="$ROOT/test/estcalibfix" +if [ ! -r "$EST_MAN" ] || [ ! -d "$EST_FIX" ]; then + no "#18: test/estcalib.manifest or test/estcalibfix missing — the calibration band cannot be measured" +else + EST_TMP="$( mktemp -d )" + cp -R "$EST_FIX" "$EST_TMP/f" + est_pins=0 + est_abs_sum=0 + while read -r label o200k cl100k pinned_est rest; do + case "$label" in ""|\#*) continue ;; esac + # shellcheck disable=SC2086 # $rest is the pinned argv: deliberately word-split, never quoted + EST_OUT="$( cd "$EST_TMP" && "$BIN" f $rest 2>/dev/null )" + EST_GOT="$( printf '%s' "$EST_OUT" | grep -oE 'est_tokens="[0-9]+"' | head -1 | grep -oE '[0-9]+' )" + if [ -z "$EST_GOT" ]; then + no "#18 $label: no est_tokens in the output — the pin says this verb prices itself" + continue + fi + est_pins=$(( est_pins + 1 )) + EST_ERR="$( python3 -c "print( round( 100.0 * ( $EST_GOT - $o200k ) / $o200k, 2 ) )" )" + EST_ABS="$( python3 -c "print( int( round( abs( 100.0 * ( $EST_GOT - $o200k ) / $o200k ) ) ) )" )" + est_abs_sum=$(( est_abs_sum + EST_ABS )) + EST_LO="$( python3 -c "print( int( $o200k * 0.75 ) )" )" + EST_HI="$( python3 -c "print( int( $o200k * 1.55 ) )" )" + if [ "$EST_GOT" -ge "$EST_LO" ] && [ "$EST_GOT" -le "$EST_HI" ]; then + ok "#18 $label: est=$EST_GOT vs o200k=$o200k (cl100k=$cl100k), ${EST_ERR}% — inside [$EST_LO,$EST_HI]" + else + no "#18 $label: est=$EST_GOT vs o200k=$o200k, ${EST_ERR}% — OUTSIDE the measured band [$EST_LO,$EST_HI]" + fi + done < "$EST_MAN" + rm -rf "$EST_TMP" + + # #18b AGGREGATE. A per-pin band tolerates one verb drifting to the edge; the MAPE is the number the + # T1 write-up quoted by hand and is what moves when the RATE TABLE is wrong rather than one emitter. + # 30 is the measured 21% plus headroom — tighten it when a calibration round earns the tightening. + if [ "$est_pins" -gt 0 ]; then + EST_MAPE=$(( est_abs_sum / est_pins )) + if [ "$EST_MAPE" -le 30 ]; then + ok "#18b est_tokens MAPE vs o200k = ${EST_MAPE}% over $est_pins pins (ceiling 30%)" + else + no "#18b est_tokens MAPE vs o200k = ${EST_MAPE}% over $est_pins pins — past the 30% ceiling" + fi + fi + + # #18c MUTATION CONTROL. Eight pins are committed; a loop that measured fewer (a manifest truncated by + # a merge, a verb that stopped printing est_tokens) asserted less than the PASS lines above suggest. + if [ "$est_pins" -ge 8 ]; then + ok "#18c $est_pins pins measured against the tokenizer manifest" + else + no "#18c only $est_pins pin(s) measured — the band arms above asserted almost nothing" + fi +fi + [ "$fail" = 0 ] && echo "ALL PASS" || echo "FAILURES ABOVE" exit $fail From 245f171c550a057f50ab0930de6ddd3e4c1decf8 Mon Sep 17 00:00:00 2001 From: joyful-ii-V-I Date: Wed, 9 Sep 2026 09:11:43 -0400 Subject: [PATCH 2/3] docs(tokenaudit): the --for compact negative is directional, so say which half is the claim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The measured magnitudes (-0.8% / -1.9%) are corpus-dependent and the self-corpus arm moved to -0.1% once this lane's own commit changed the tree it ranks. Both readings are recorded with the ref they were taken at, and the sentence now carries the DIRECTION as the claim — a wash or a small loss on --for, never the 39-66% token saving the navigation verbs show — because a number that moves with the corpus is not the thing a reader should carry away. Co-Authored-By: Claude Opus 5 --- bench/tokenaudit/README.md | 11 +++++++---- docs/EVALS.md | 9 ++++++--- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/bench/tokenaudit/README.md b/bench/tokenaudit/README.md index cbaac67d..c2a73f7b 100644 --- a/bench/tokenaudit/README.md +++ b/bench/tokenaudit/README.md @@ -83,10 +83,13 @@ legend's token share ran 3.2% (whole map) to **79.8% (`--callees`)**, with `--ed started this (callstack/agent-device #2400, "a fixed per-call preamble, 62% of `--callers`' whole response") while placing `--callers` itself at 42.9%/33.3% on the two corpora here. -**4. `--legend=compact` is not a saving on `--for`.** Measured −0.8% (this repo) and −1.9% (the C++ tree): -the compact posture emitted *more* tokens, because `--for` is budget-shaped and the bytes the legend frees -are refilled from the trim ladder's tail. `--help`'s advice ("MAKING REPEATED CALLS? USE compact") is right -for the navigation verbs and wrong-signed for the bundle it also names as "a little". +**4. `--legend=compact` is not a saving on `--for`.** Measured −0.8% (this repo) and −1.9% (the C++ tree) +at `4c10be9d`: the compact posture emitted *more* tokens, because `--for` is budget-shaped and the bytes the +legend frees are refilled from the trim ladder's tail. Re-measured after the commit that added this +directory, the same two arms read −0.1% and −1.9% — the magnitude moves with the corpus, the direction is +what to carry: a wash or a small loss, never the 39-66% token saving the navigation verbs show. `--help`'s +advice ("MAKING REPEATED CALLS? USE compact") is right for the navigation verbs and wrong-signed for the +bundle it also names as "a little". **5. What a `--token-budget=N` actually delivers.** Real o200k tokens as a fraction of the requested N, and the reported `est_tokens` error at that budget: diff --git a/docs/EVALS.md b/docs/EVALS.md index 33cfcf15..1fb19ed7 100644 --- a/docs/EVALS.md +++ b/docs/EVALS.md @@ -12905,9 +12905,12 @@ whole response"); `--callers` itself measures 42.9% / 33.3% on the two corpora h smaller answer than either, and the *shape* of their finding reproduces. One negative worth recording: **`--legend=compact` is not a saving on `--for`.** Measured −0.8% and −1.9% -in tokens — the compact posture emitted *more*, because `--for` is budget-shaped and the bytes the legend -frees are refilled from the trim ladder's tail. `--help`'s advice is right for the navigation verbs and -wrong-signed for the bundle it also names. +in tokens at `4c10be9d` — the compact posture emitted *more* — because `--for` is budget-shaped and the +bytes the legend frees are refilled from the trim ladder's tail. Re-measured after this section's own +commit the same two arms read −0.1% and −1.9%, so the magnitude moves with the corpus and the DIRECTION is +what to carry: on `--for` the compact posture is a wash or a small loss, never the 39-66% token saving the +navigation verbs show. `--help`'s advice is right for the navigation verbs and wrong-signed for the bundle +it also names. ### 3. The loop the per-call number cannot see (`bench/tokenaudit/loop_ledger.py`) From 6d878e5f8b824a927c79acec29f21eccbc8e246f Mon Sep 17 00:00:00 2001 From: joyful-ii-V-I Date: Wed, 9 Sep 2026 09:34:13 -0400 Subject: [PATCH 3/3] =?UTF-8?q?fix(tokenaudit):=20the=20results=20file=20i?= =?UTF-8?q?s=20public,=20so=20the=20corpora=20are=20labels=20=E2=80=94=20r?= =?UTF-8?q?ipwirepubliccheck=20was=20right?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first committed results JSON carried all three leak classes at once: an absolute home path in `bin` and two `root` values, the private development tree's NAME as a corpus key, and the personal identifier inside those paths. test/ripwirepubliccheck.sh caught every one on CI shard 3/4, on all four Linux flavours, which is the gate working exactly as designed — a calibration sweep WANTS the corpora that must not appear in a public repository, so this class was going to happen the moment the instrument was committed rather than run. Fixed FORWARD, per the standing directive, and fixed at the SHAPE rather than the instance (CONTRIBUTING §2 rule 5): sweep.py no longer writes a path at all. The caller's LABEL is the corpus identity (`root_label`), `bin` is a basename, and the file carries a `redaction` key saying so, so a reader knows the omission is deliberate and that no measurement was altered. The committed results keep every number exactly as taken; only the identifying strings changed, and the second corpus is now named by its shape ("a private 1500-file C++ tree"), which is how docs/EVALS.md and the README already referred to it. Two smaller corrections in the same file: the header still named a `test/estcalibcheck.sh` that was never created — the arm was absorbed into test/tokenbudgetcheck.sh #18 — and the new redaction comment tripped arm 2 by quoting the very path prefix it was warning about. Co-Authored-By: Claude Opus 5 --- bench/tokenaudit/README.md | 4 ++++ .../results/tokenaudit-2026-09-09.json | 9 +++++---- bench/tokenaudit/sweep.py | 18 +++++++++++++----- 3 files changed, 22 insertions(+), 9 deletions(-) diff --git a/bench/tokenaudit/README.md b/bench/tokenaudit/README.md index c2a73f7b..21f55bf3 100644 --- a/bench/tokenaudit/README.md +++ b/bench/tokenaudit/README.md @@ -19,6 +19,10 @@ The split is the point. G3 is one deterministic build step with nothing host-ins imported tiktoken would be a dependency the build contract forbids. The tokenizer runs by hand and writes numbers; the gate reads numbers. Same shape as `test/printf_parity.manifest`. +The corpus **labels** are what the results JSON records — never a path, never a private tree's name. +`test/ripwirepubliccheck.sh` caught all three leak classes in this directory's first committed results +file; `sweep.py` now writes labels by construction so the gate has nothing to catch. + ```bash python3 -m venv /tmp/tokvenv && /tmp/tokvenv/bin/pip install tiktoken /tmp/tokvenv/bin/python bench/tokenaudit/sweep.py --bin build/ripwire \ diff --git a/bench/tokenaudit/results/tokenaudit-2026-09-09.json b/bench/tokenaudit/results/tokenaudit-2026-09-09.json index d1d5ecd2..696bf350 100644 --- a/bench/tokenaudit/results/tokenaudit-2026-09-09.json +++ b/bench/tokenaudit/results/tokenaudit-2026-09-09.json @@ -1,9 +1,9 @@ { - "bin": "/Users/qgames/AppDevelopLocal/project2/rw-hv-B/build/ripwire", + "bin": "build/ripwire", "corpora": { - "canyonraid48": { + "private-cpp-tree": { "file": "math/doctest.h", - "root": "/Users/qgames/AppDevelopLocal/project2/canyonraid48", + "root_label": "a private 1500-file C++ tree", "rows": [ { "argv": [], @@ -450,7 +450,7 @@ }, "self": { "file": "src/infra/svector.h", - "root": "/Users/qgames/AppDevelopLocal/project2/rw-hv-B", + "root_label": "this repository", "rows": [ { "argv": [], @@ -901,6 +901,7 @@ "symbol": "buf" } }, + "redaction": "corpus roots and the binary path are recorded as LABELS, never as absolute paths, and the second corpus is named by its shape rather than its name: it is a private tree and this file is public. bench/tokenaudit/sweep.py writes it this way; nothing else was altered, every count below is the measurement as taken.", "schema": "ripwire.tokenaudit/v1", "version": "ripwire 0.5.0 (dev, AppleClang 21.0.0.21000101, emit=std::print, built_from=4c10be9d7)" } diff --git a/bench/tokenaudit/sweep.py b/bench/tokenaudit/sweep.py index c3c10b0d..287b8fdf 100644 --- a/bench/tokenaudit/sweep.py +++ b/bench/tokenaudit/sweep.py @@ -25,9 +25,10 @@ # WHAT IT IS NOT. Not a gate. tiktoken is not a build dependency and never will be (G3: one # deterministic build step, nothing host-installed), and o200k_base is not Claude's tokenizer — # it is the closest public stand-in, which is exactly why kTokenCalib was calibrated against it and -# why a gate that needs it at runtime would be a dependency problem. The gate this round adds -# (test/estcalibcheck.sh) reads PINNED counts out of a manifest this script writes; re-running this -# script is how the manifest is regenerated, deliberately by hand. +# why a gate that needs it at runtime would be a dependency problem. The gate arm this round adds +# (test/tokenbudgetcheck.sh #18 — absorbed into the gate that already owns est_tokens rather than added +# beside it) reads PINNED counts out of test/estcalib.manifest, which bench/tokenaudit/pin.py writes; +# regenerating that manifest is a deliberate manual step, never a build step. # # Usage: # python3 bench/tokenaudit/sweep.py --bin build/ripwire --out bench/tokenaudit/results/x.json \ @@ -142,7 +143,14 @@ def main(): encs = {"o200k": tiktoken.get_encoding("o200k_base"), "cl100k": tiktoken.get_encoding("cl100k_base")} - result = {"bin": binpath, "corpora": {}, "schema": "ripwire.tokenaudit/v1"} + # REDACTION IS BY CONSTRUCTION, not by review. This file's output is COMMITTED, and the corpora a + # calibration run wants are exactly the ones that must not appear in a public repository: someone's + # home directory, and a private tree whose NAME is the leak. test/ripwirepubliccheck.sh caught all + # three classes (an absolute home path, a private tree's name, a personal identifier) on the first run of + # this script, in the results JSON. So no path is written at all: the caller's LABEL is the identity, + # and a label that names a private tree is the caller's to choose well. + result = {"bin": os.path.basename(binpath), "corpora": {}, "schema": "ripwire.tokenaudit/v1", + "redaction": "corpus roots and the binary path are labels, never paths — this file is public"} ver, _, _ = run(binpath, ".", ["--version"]) result["version"] = ver.decode("utf-8", "replace").strip().splitlines()[0] if ver else "" @@ -182,7 +190,7 @@ def main(): else: row["compact_refused"] = cerr.strip()[:200] rows.append(row) - result["corpora"][name] = {"root": path, "symbol": sym, "file": fil, "rows": rows} + result["corpora"][name] = {"root_label": name, "symbol": sym, "file": fil, "rows": rows} os.makedirs(os.path.dirname(os.path.abspath(args.out)), exist_ok=True) with open(args.out, "w") as f: