From 4b23f7eb9cf9dc1800bac03841e1c5035f9fe475 Mon Sep 17 00:00:00 2001 From: joyful-ii-V-I <210578314+joyful-ii-V-I@users.noreply.github.com> Date: Wed, 12 Aug 2026 10:02:50 -0400 Subject: [PATCH] feat(meter): the report answers whether a verb ENDED the question MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An output only saves tokens if it terminates the question that prompted it; one that spawns the next command is net-additive. bench/substitution_report.py gains §5, per-verb terminality, computed from the meter log it already reads: for each ripwire call, look ahead inside its session -- to the next ripwire call, the session end, or 5 calls, whichever comes first -- and ask whether a sweep-class call followed. Verb, n, terminal%, and the first follow-up class that broke it. Three definitional choices are the substance, and each is printed above the table rather than left to the docs, because a terminality percentage read without its window rule is a number somebody quotes wrong: - the sweep set is WIDER than §1's `native` family. git diff/log/show --stat are history retrieval, and a map followed by a raw git-history sweep did not terminate the question any more than a map followed by grep did. They stay out of the substitution ratio, where they are a different QUESTION rather than a different tool for the same one, and they count here, where the question is whether the answer landed; - an EMPTY window counts terminal -- no sweep happened -- which is the definition's softest spot, so the count of them is disclosed under the table instead of folded in silently; - n travels with every percentage, and a verb under n=10 carries an explicit NOTE row. Not a significance test; a floor under the reader. The verb itself is read lexically from `detail`: the ripwire word by basename (behind `cd X && VAR=y`, at any path), then the first flag before any pipe or redirect, so a pipeline's own flags are never mistaken for the verb. The skip list of verb-agnostic options is deliberately NOT a mirror of the binary's ~70-row dispatch table -- a mirror rots silently, and being wrong here costs one row shown by name under a modifier rather than one hidden. The 200-character `detail` cap is named in all three of its forms rather than guessed at, because a real log hit two of them on the first reading: cut before the ripwire word is `(unparsed)`, cut after it with no flag yet is `(truncated)`, and cut mid-flag keeps the flag with a trailing `...`. Filing a half-written flag under the prefix that survived would split one verb's n across two rows and understate both. Gate: test/hookcheck.sh arms T1-T12, on a synthetic fixture log whose answer is known by construction -- a terminal --for, a non-terminal one followed by three greps, windows truncated by the next ripwire call and by session end, the k=5 edge asserted from BOTH sides (five non-sweep calls put a grep out of the window, four leave it in), a git-history follow-up, an n<10 verb with its NOTE and an n>=10 verb without one, an MCP row, a flagless map behind a cd prefix and ahead of a pipeline, and both truncation labels. Exact table values, not "it did not crash": 11 of the 12 were red against the previous script. The gate's own sink cannot serve as the fixture -- its contents change with every arm anyone adds above it. Analysis-side only: no schema change, no hook change, and the levels stay in the operator-local ledger. 397 gates green. --- bench/substitution_report.py | 182 +++++++++++++++++++++++++++++++++++ docs/SUBSTITUTION_METER.md | 98 +++++++++++++++++-- test/hookcheck.sh | 152 +++++++++++++++++++++++++++++ 3 files changed, 425 insertions(+), 7 deletions(-) diff --git a/bench/substitution_report.py b/bench/substitution_report.py index 63ed2835..3f154440 100755 --- a/bench/substitution_report.py +++ b/bench/substitution_report.py @@ -32,17 +32,172 @@ # trigrams in seq order. A chain that recurs is a candidate SCENARIO for one verb to absorb whole. # Counts only — which chain is worth absorbing is the S4 survey's judgment, not this script's. # +# TERMINALITY (§5, Track T item T0). An output only saves tokens if it ENDS the question that prompted +# it; one that spawns a sweep is net-additive. §5 turns that into a per-verb count: for each ripwire +# call, look ahead within its session and ask whether a native-search call followed. The window and +# the sweep set are stated in the section's own header and in docs/SUBSTITUTION_METER.md, because a +# terminality number is meaningless without them. Counts and ratios only, as everywhere else here — +# and never a percentage without the n beside it, with an explicit NOTE row under any verb whose n is +# too small to read as a rate. +# # Usage: # python3 bench/substitution_report.py [~/.ripwire/substitution.jsonl] [--top N] [--tag REPO] import argparse import collections import json import os +import re import sys RIPWIRE_FAMILY = "ripwire" NATIVE_FAMILY = "native" +# ── §5 terminality: the three constants the metric is made of ─────────────────────────────────────── +# THE SWEEP SET is wider than the `native` family on purpose. `git diff`/`git log`/`git show --stat` +# are history RETRIEVAL — a map followed by a raw git-history sweep did not terminate the question any +# more than a map followed by grep did — so the git-history classes count here even though they are +# deliberately outside §1's substitution ratio, where they are a different QUESTION rather than a +# different tool for the same one. The state-changing git classes (`git-misc`, `git-remote`) are not +# retrieval and are not in the set. +SWEEP_CLASSES = frozenset(("grep", "read", "glob", "find", "git-diff", "git-log", "git-show-stat")) +# The look-ahead is capped so a ripwire call is never blamed for a sweep half a session later. +TERMINALITY_WINDOW = 5 +# Below this an n is printed with a NOTE instead of being read as a rate. Not a significance test — +# there is no test to run on a non-randomized single-operator log — just a floor under the reader. +SMALL_N = 10 + +# A verb-agnostic OPTION is skipped when scanning a command line for the verb. This list is small, +# lexical and deliberately NOT a mirror of the binary's dispatch table (src/main.cpp +# scanReportVerbPrecedence): a mirror of ~70 verbs would rot silently, and the cost of being wrong +# here is one row attributed to a modifier, which the table shows by name rather than hiding. A +# modifier this list does not know is reported AS the verb — that is the disclosure, not a claim of +# completeness. +NON_VERB_FLAGS = frozenset(( + "--no-cache", "--cache-dir", "--no-route", "--route", "--token-budget", "--top-k", "--rank-by", + "--stable", "--metrics", "--format", "--exclude", "--include", "--jobs", "--lang", "--no-color", +)) + +MCP_PREFIX = "mcp__ripwire__" +# The hook caps `detail` at 200 characters, so a long command line can be cut mid-flag. Such a row is +# LABELLED as truncated rather than counted under whatever prefix survived: silently filing `--qualit` +# apart from `--quality-delta` would split one verb's n across two rows and understate both. +DETAIL_CAP = 200 +FLAG_RE = re.compile(r"^(--[A-Za-z0-9][A-Za-z0-9-]*)") +# Where the ripwire command ENDS: a pipe, a redirect (`>`, `2>`, `2>&1`), a separator. Flags after one +# of these belong to some other program — `ripwire . | grep -n --color foo` is a map, not a --color. +BREAK_RE = re.compile(r"^(?:[0-9]*[<>]|[|;&])") + + +def ripwire_token(toks): + """Index of the word that IS the ripwire command, or None. Matched by basename, so + `./build/ripwire`, an absolute path and a bare `ripwire` all hit, and a `cd X && VAR=y` prefix in + front of it is simply skipped over rather than parsed.""" + for i, tok in enumerate(toks): + if os.path.basename(tok.strip("'\"")) == "ripwire": + return i + return None + + +def ripwire_verb(row): + """The verb a ripwire-family row asked for, as a label. MCP rows carry it in the tool name; CLI + rows carry a whole command line in `detail`, so the ripwire word is located (above) and the first + flag after it — before any shell break — is the verb. A flagless run is the core map. + + Three ways the 200-character `detail` cap defeats that, each named rather than guessed at: cut + before the ripwire word at all is `(unparsed)`; cut after it with no flag and no shell break yet + is `(truncated)`, since the verb may be just past the cap; and cut in the MIDDLE of the flag + yields that flag with a trailing `...`. A complete flag that happens to end a 200-character line + is indistinguishable from a cut one and is marked the same way — the label errs toward saying so. + """ + tool = str(row.get("tool") or "") + if tool.startswith(MCP_PREFIX): + return "mcp:" + (tool[len(MCP_PREFIX):] or "?") + detail = str(row.get("detail") or "") + toks = detail.split() + capped = len(detail) >= DETAIL_CAP + start = ripwire_token(toks) + if start is None: + return "(unparsed)" + for i in range(start + 1, len(toks)): + if BREAK_RE.match(toks[i]): + return "(map)" # the ripwire command ended at a pipe/redirect: it took no verb + m = FLAG_RE.match(toks[i]) + if m and m.group(1) not in NON_VERB_FLAGS: + return m.group(1) + ("..." if capped and i == len(toks) - 1 else "") + return "(truncated)" if capped else "(map)" + + +def session_order(rows): + """Sessions -> their rows in `seq` order. `session-start` rows are boundaries, not calls, and are + dropped here for the same reason §4 drops them: they are not a thing an agent chose to run.""" + sessions = collections.defaultdict(list) + for i, r in enumerate(rows): + if r.get("class") == "session-start": + continue + sessions[str(r.get("session"))].append((r.get("seq", 0), i, r)) + out = [] + for _sid, rs in sorted(sessions.items(), key=lambda kv: str(kv[0])): + rs.sort(key=lambda t: (t[0], t[1])) + out.append([t[2] for t in rs]) + return out + + +def window_verdict(seq_rows, idx): + """(follow-up class or None, calls seen) for the window after the ripwire call at `idx`. + + The window is the calls after it up to — whichever comes first — the next ripwire call, the end + of the session, or TERMINALITY_WINDOW calls. `None` means TERMINAL: no sweep-class call appeared + in it. Otherwise the FIRST sweep-class call is the follow-up — three greps in a row are one + follow-up, not three.""" + seen = 0 + for nxt in seq_rows[idx + 1:idx + 1 + TERMINALITY_WINDOW]: + if nxt.get("family") == RIPWIRE_FAMILY: + break + seen += 1 + if nxt.get("class") in SWEEP_CLASSES: + return str(nxt.get("class")), seen + return None, seen + + +def terminality(rows): + """Per-verb (n, terminal, follow-up counter), plus the number of EMPTY windows. + + An empty window — the very next call was another ripwire call, or the session ended — is TERMINAL + by the definition above, since no sweep happened. That is the definition's softest spot, so the + count is returned and printed rather than folded in silently: a run of consecutive ripwire calls + manufactures terminal windows, and the disclosure is what lets a reader discount them.""" + stats = {} + empty = 0 + for seq_rows in session_order(rows): + for idx, r in enumerate(seq_rows): + if r.get("family") != RIPWIRE_FAMILY: + continue + st = stats.setdefault(ripwire_verb(r), [0, 0, collections.Counter()]) + st[0] += 1 + follow, seen = window_verdict(seq_rows, idx) + if seen == 0: + empty += 1 + if follow is None: + st[1] += 1 + else: + st[2][follow] += 1 + return stats, empty + + +def top_followup(counter): + """The commonest follow-up class, ties broken alphabetically so the table is deterministic.""" + if not counter: + return "(none)" + cls, n = sorted(counter.items(), key=lambda kv: (-kv[1], kv[0]))[0] + return "%s (%d)" % (cls, n) + + +def terminality_row(verb, n, term, counter): + print(" %-24s %6d %10s %s" + % (verb, n, "%.1f%%" % (100.0 * term / n) if n else "n/a", top_followup(counter))) + if 0 < n < SMALL_N: + print(" NOTE: n=%d (<%d) -- too few calls to read as a rate" % (n, SMALL_N)) + def load(path): """Rows, in file order. A malformed line is counted and skipped — never fatal, never silent.""" @@ -220,6 +375,33 @@ def main(): for gram, c in sorted(grams.items(), key=lambda kv: (-kv[1], kv[0]))[:args.top]: print(" %-46s %6d" % (" -> ".join(gram), c)) + # ── §5 terminality by verb ────────────────────────────────────────────────────────────────────── + # §4 asks which chains recur; this asks the sharper question underneath it — of the chains that + # START with ripwire, how many END there. The definitions are printed above the table rather than + # left to the docs, because a terminality percentage read without its window rule is a number + # somebody will quote wrong. The reader gets counts; which verb to enrich is Track T's judgment. + section("5. terminality by verb (did the output END the question, or spawn a sweep?)") + print(" window : the calls after a ripwire call, up to the next ripwire call, the session end,") + print(" or %d calls -- whichever comes first" % TERMINALITY_WINDOW) + print(" TERMINAL : no sweep-class call in the window. sweep = %s" + % " ".join(sorted(SWEEP_CLASSES))) + print("") + print(" %-24s %6s %10s %s" % ("verb", "n", "terminal%", "top follow-up")) + stats, empty = terminality(rows) + if not stats: + print(" (no ripwire-family calls in this log -- nothing to measure)") + return 0 + for verb, (n, term, counter) in sorted(stats.items(), key=lambda kv: (-kv[1][0], kv[0])): + terminality_row(verb, n, term, counter) + allc = collections.Counter() + for _v, (_n, _t, counter) in stats.items(): + allc.update(counter) + terminality_row("(all)", sum(s[0] for s in stats.values()), sum(s[1] for s in stats.values()), allc) + print("") + print(" empty windows: %d of %d -- the next observed call was another ripwire call, or the" + % (empty, sum(s[0] for s in stats.values()))) + print(" session ended. These count TERMINAL by the definition above.") + return 0 diff --git a/docs/SUBSTITUTION_METER.md b/docs/SUBSTITUTION_METER.md index ef5e2048..9782c9ea 100644 --- a/docs/SUBSTITUTION_METER.md +++ b/docs/SUBSTITUTION_METER.md @@ -406,20 +406,104 @@ python3 bench/substitution_report.py -h # repo filter, row l Scrub first if the log predates 2026-08-12 — it may carry gate fixtures (above), and the report has no way to tell them from real rows. -Stdlib only, deterministic, counts only. Four sections: the rate split by nudge exposure and by arm; -the class composition of both sides; a per-repo breakdown; and **within-session class n-grams** -(bigrams and trigrams in `seq` order). That last section is the one to watch — the hypothesis worth -testing is that the real opportunity is a recurring *chain* (`grep → read → read`) that a single verb -could absorb whole, not any single call. The script prints counts and no interpretation; which chain -is worth absorbing is a judgment made elsewhere. +Stdlib only, deterministic, counts only. Five sections: the rate split by nudge exposure and by arm; +the class composition of both sides; a per-repo breakdown; **within-session class n-grams** (bigrams +and trigrams in `seq` order); and **terminality by verb** (below). The n-gram section is the one to +watch — the hypothesis worth testing is that the real opportunity is a recurring *chain* (`grep → +read → read`) that a single verb could absorb whole, not any single call. The script prints counts +and no interpretation; which chain is worth absorbing is a judgment made elsewhere. + +## Terminality (§5) + +An output only saves tokens if it **ends** the question that prompted it. One that spawns the next +command is net-additive: the map is paid for and then the sweep is paid for as well. §4 asks which +chains recur; §5 asks the sharper question underneath it — of the chains that *start* with ripwire, +how many end there. It is the per-verb instrument the enrichment work is ranked by, and it doubles as +a release-over-release ledger for output quality. + +### The definition, in full + +For every **ripwire-family** row in the log (`ripwire-cli` and `ripwire-mcp`), look ahead inside its +own session, in `seq` order: + +| | | +| --- | --- | +| **Window** | The calls after it, up to — whichever comes **first** — the next ripwire call, the end of the session, or **5** calls. | +| **TERMINAL** | No *sweep-class* call appears in that window. | +| **Follow-up** | When it is not terminal, the **first** sweep-class call in the window is recorded as the follow-up class. Three greps in a row contribute one follow-up, not three. | +| **Sweep set** | `grep` `read` `glob` `find` — and `git-diff` `git-log` `git-show-stat`. | +| **Excluded** | `session-start` rows are session boundaries, not calls an agent chose to run, and are dropped before windowing (the same rule §4 uses). | + +Two of those lines are choices worth defending rather than defaults. + +**The sweep set is wider than the `native` family.** `git diff` / `git log` / `git show` with a stat summary are +history *retrieval*: a map followed by a raw git-history sweep did not terminate the question any +more than a map followed by `grep` did. They stay outside §1's substitution ratio — there they are a +different **question**, not a different tool for the same one — and they count here, where the +question is whether the answer landed. The state-changing git classes (`git-misc`, `git-remote`) are +not retrieval and are in neither. + +**An empty window counts as TERMINAL** — the next observed call was another ripwire call, or the +session ended, so no sweep happened. That is the definition applied honestly, and it is also the +definition's softest spot, so the count of empty windows is **printed under the table** instead of +being folded in silently. A run of consecutive ripwire calls manufactures terminal windows; the +disclosure is what lets a reader discount them. + +### Which verb a row asked for + +An MCP row carries its verb in the tool name (`mcp__ripwire__for` → `mcp:for`). A CLI row carries a +whole command line in `detail`, so the verb is read lexically: find the leading word whose basename +is `ripwire` — it may be `./build/ripwire`, an absolute path, or sit behind `cd X && VAR=y` — then +take the **first flag after it**, stopping at the first pipe, redirect or separator so a pipeline's +own flags are never mistaken for the verb. A run with no flag is the core map, `(map)`. + +The 200-character `detail` cap defeats that in three ways, and each is **named rather than guessed +at** — a real log hit two of them on the metric's first reading: + +| Cut | Label | +| --- | --- | +| before the ripwire word | `(unparsed)` | +| after it, with no flag and no shell break yet — the verb may be one character past the cap | `(truncated)` | +| in the middle of the flag | that flag with a trailing `...` | + +The third case is why the marking exists at all: a half-written flag filed under the prefix that +survived would split one verb's n across two rows and understate both. A *complete* flag that happens +to end a 200-character line cannot be told from a cut one, and is marked the same way — the label +errs toward saying so. + +A short list of **verb-agnostic options** (`--no-cache`, `--token-budget`, `--top-k`, `--rank-by`, +`--format`, `--exclude`, …) is skipped while scanning. That list is deliberately *not* a mirror of +the binary's ~70-row dispatch table: a mirror would rot silently, and the cost of being wrong here is +one row attributed to a modifier — which the table then shows by name, under that modifier, rather +than hiding. + +### Small n is stated, never smoothed + +Every row prints its **n** beside its percentage, and a verb whose n is under **10** gets an explicit +`NOTE: n=… (<10)` row under it. This is not a significance test — there is no test to run on a +non-randomized single-operator log — it is a floor under the reader. A terminality percentage read +without its n and its window rule is a number somebody will quote wrong. + +As everywhere else in this document, the mechanism is public and the **levels are not**: concrete +readings are operator telemetry and live in the local ledger, not in the repository. ## Gate -`test/hookcheck.sh` section (11), arms M1–M27b: a row is written at the default global path with the +`test/hookcheck.sh` section (11), arms M1–M27b and T1–T11: a row is written at the default global path with the full field set; the rtk unwrap; the `unclassified` fallback; out-of-scope calls writing no row; the `nudged`/`dedup`/`post_nudge`/`post_sweep`/`seq` fields; both arms; one global log across two repos; and — the arm that matters most — an unwritable log costing the hooked command nothing. +Arms **T1–T11** pin §5 against a **synthetic fixture log** whose right answer is known by +construction — a terminal `--for`, a non-terminal one followed by three greps, windows truncated by +the next ripwire call and by session end, the k=5 window edge asserted from *both* sides (five +non-sweep calls put a grep out of the window, four leave it in), a git-history follow-up, an n<10 +verb with its NOTE row and an n≥10 verb without one, an MCP row, and a flagless map behind a `cd` +prefix and ahead of a pipeline carrying flags of its own. The assertions are exact table values, not +"it did not crash": every count, percentage and follow-up in the fixture's table is spelled out in +the gate. The gate's own sink cannot serve here — its contents change with every arm anyone adds +above it, so nothing about it is known in advance. + Section (12), arms S1–S14, covers the sweep escalation and the widened classifier: three same-class calls produce exactly one escalation carrying the paste-ready command; two do not; a fourth is silent; the classes dedup independently; the row carries `nudge":"sweep3"` and the next row diff --git a/test/hookcheck.sh b/test/hookcheck.sh index 2216eff6..79dd77a2 100755 --- a/test/hookcheck.sh +++ b/test/hookcheck.sh @@ -658,9 +658,161 @@ if [ -f "$REPORT" ]; then grep -qiE 'bigram|n-gram|ngram' "$TMP/report.out" \ && ok "M27b meter: the report prints within-session command-class n-grams (scenario bundles)" \ || no "M27b meter: the report has no n-gram section" +fi + +# ── T1-T11: §5 terminality by verb, pinned on a SYNTHETIC fixture log (Track T item T0) ──────────── +# +# WHY A FIXTURE AND NOT THE GATE'S OWN SINK. Every other meter arm here asserts a property of the +# HOOK, so the hook's own output is the right input. §5 is an ANALYSIS, and an analysis is gated by +# feeding it data whose right answer is known by construction — the sink's contents change with every +# arm anyone adds above, so an assertion against it could only ever be "it did not crash". +# +# The fixture below encodes, by hand, one instance of each thing the metric has to get right: +# +# - a TERMINAL --for (t-a: the follow-ups are build/git-misc/shell-misc, none of them a sweep); +# - a NON-TERMINAL --for followed by THREE greps (t-b), so the follow-up recorded is the FIRST one; +# - a window truncated by the NEXT RIPWIRE CALL (t-a seq4) and by SESSION END (t-a seq6, t-d seq7, +# t-e seq1) — the empty-window disclosure counts exactly those; +# - the k=5 EDGE, twice and from both sides (t-f, t-g): five non-sweep calls then a grep is TERMINAL +# because the grep is out of the window; four then a grep is not. An off-by-one in either +# direction reds exactly one of that pair, which is the point of gating the edge rather than the +# middle; +# - a git-history follow-up (t-b seq8), which counts as a sweep here though it is outside §1's ratio; +# - an n<10 verb (--grep, n=3) carrying its NOTE row, and an n>=10 verb (--for, n=12) carrying none; +# - an MCP row, whose verb comes from the tool name rather than from a command line; +# - the flagless map behind a `cd` prefix and ahead of a pipeline (t-e) — `| grep -n --color foo` +# must NOT be read as the verb `--color`. +# +# The assertions are on the space-SQUEEZED table rows, so they pin every value the table states while +# leaving column widths free to be laid out for a reader. +TERMLOG="$TMP/terminality.jsonl" +termrow() +{ + # termrow SEQ SESSION TOOL CLASS FAMILY DETAIL + printf '{"v":2,"ts":"2026-08-12T00:00:00Z","seq":%s,"session":"%s","repo":"/x/repo","tag":"repo","tool":"%s","class":"%s","family":"%s","nudged":0,"nudge":"none","post_nudge":0,"post_sweep":0,"arm":"treatment","detail":"%s"}\n' \ + "$1" "$2" "$3" "$4" "$5" "$6" >>"$TERMLOG" +} +termrip() { termrow "$1" "$2" Bash ripwire-cli ripwire "$3"; } +termbuild() { termrow "$1" "$2" Bash build other 'cmake --build build -j'; } +termgrep() { termrow "$1" "$2" Grep grep native 'needle'; } +termread() { termrow "$1" "$2" Read read native '/x/repo/a.cpp'; } + +: >"$TERMLOG" +# t-a — three TERMINAL --for: non-sweep follow-ups, then a window ended by the next ripwire call, then +# a window ended by the session. +termrip 1 t-a './build/ripwire . --for=alpha' +termbuild 2 t-a +termrow 3 t-a Bash git-misc git 'git status' +termrip 4 t-a './build/ripwire . --for=beta' +termrow 5 t-a Bash shell-misc other 'ls -la docs/' +termrip 6 t-a './build/ripwire . --for=gamma' +# t-b — the sweep case: --for then THREE greps (first one is the recorded follow-up), --for then one +# grep, --for then a git-history call. +termrip 1 t-b 'cd /x/repo && ./build/ripwire . --for=delta' +termgrep 2 t-b; termgrep 3 t-b; termgrep 4 t-b +termrip 5 t-b './build/ripwire . --for=epsilon --top-k=3' +termgrep 6 t-b +termrip 7 t-b './build/ripwire . --for=zeta' +termrow 8 t-b Bash git-log git 'git log --oneline -20' +# t-c — the --grep verb (n<10, two read follow-ups) and the MCP verb. +termrip 1 t-c './build/ripwire . --grep=needle' +termread 2 t-c +termrip 3 t-c './build/ripwire . --grep=needle' +termread 4 t-c +termrip 5 t-c './build/ripwire . --grep=needle' +termbuild 6 t-c +termrow 7 t-c mcp__ripwire__for ripwire-mcp ripwire 'rank the parser' +termbuild 8 t-c +termrow 9 t-c mcp__ripwire__for ripwire-mcp ripwire 'rank the ranker' +# t-d — four more TERMINAL --for, which is what lifts --for over the small-n floor. +termrip 1 t-d './build/ripwire . --for=iota' +termbuild 2 t-d +termrip 3 t-d './build/ripwire . --for=kappa' +termrow 4 t-d Bash shell-misc other 'ls -la' +termrip 5 t-d './build/ripwire . --for=lambda' +termrow 6 t-d Bash git-misc git 'git status' +termrip 7 t-d './build/ripwire . --for=mu' +# t-e — the flagless map, behind a cd prefix, ahead of a pipeline carrying flags of its own. +termrip 1 t-e 'cd /x/repo && ./build/ripwire . > /tmp/a 2>&1 | grep -n --color foo' +# t-f/t-g — the k=5 window edge, from both sides. FIVE non-sweep calls put the grep out of the window +# (terminal); FOUR leave it inside (not terminal). +termrip 1 t-f './build/ripwire . --for=nu' +termbuild 2 t-f; termbuild 3 t-f; termbuild 4 t-f; termbuild 5 t-f; termbuild 6 t-f +termgrep 7 t-f +termrip 1 t-g './build/ripwire . --for=xi' +termbuild 2 t-g; termbuild 3 t-g; termbuild 4 t-g; termbuild 5 t-g +termgrep 6 t-g + +if [ -f "$REPORT" ]; then + python3 "$REPORT" "$TERMLOG" >"$TMP/term.out" 2>&1; RCT=$? + tr -s ' ' <"$TMP/term.out" >"$TMP/term.sq" + echo "-- substitution_report.py §5 on the terminality fixture --" + sed -n '/^5\./,$p' "$TMP/term.out" + termhas() + { + # termhas ARMID EXPECTED-SQUEEZED-LINE DESCRIPTION + grep -Fqx " $2" "$TMP/term.sq" \ + && ok "$1 terminality: $3" \ + || no "$1 terminality: expected the row [$2] — see $TMP/term.out" + } + [ "$RCT" -eq 0 ] && grep -qi 'terminality by verb' "$TMP/term.out" \ + && ok "T1 terminality: the report prints a §5 terminality-by-verb section" \ + || no "T1 terminality: exit=$RCT and/or no §5 section — see $TMP/term.out" + # The definitions travel WITH the number: a terminality % read without its window rule is a + # number somebody quotes wrong, so the header is gated, not left to the docs. + grep -q 'up to the next ripwire call, the session end' "$TMP/term.out" \ + && grep -q 'or 5 calls' "$TMP/term.out" \ + && grep -q 'sweep = find git-diff git-log git-show-stat glob grep read' "$TMP/term.out" \ + && ok "T2 terminality: §5 states its window and its sweep set above the table" \ + || no "T2 terminality: §5 does not state the window/sweep definitions — see $TMP/term.out" + termhas T3 "--for 12 66.7% grep (3)" "--for: 12 calls, 8 terminal, first follow-up grep x3" + termhas T4 "--grep 3 33.3% read (2)" "--grep: 3 calls, 1 terminal, read x2" + termhas T5 "mcp:for 2 100.0% (none)" "an MCP row takes its verb from the tool name" + termhas T6 "(map) 1 100.0% (none)" "a flagless run behind cd, ahead of a pipeline, is the map" + termhas T7 "(all) 18 66.7% grep (3)" "the (all) row totals every verb" + termhas T8 "empty windows: 4 of 18 -- the next observed call was another ripwire call, or the" \ + "the empty-window count is disclosed, not folded in silently" + # n<10 gets a NOTE; n>=10 does not. Three verbs are under the floor (--grep, mcp:for, (map)); --for + # and (all) are over it. A NOTE on every row would be as useless as a NOTE on none. + NOTES="$( grep -c 'NOTE: n=. (<10)' "$TMP/term.sq" )"; [ -n "$NOTES" ] || NOTES=0 + [ "$NOTES" -eq 3 ] \ + && ok "T9 terminality: exactly the three n<10 verbs carry a small-n NOTE row" \ + || no "T9 terminality: $NOTES NOTE row(s), expected 3 — see $TMP/term.out" + # The k=5 edge, asserted as the PAIR. One arm alone passes under an off-by-one in one direction. + : >"$TMP/term_f.jsonl"; grep -F '"t-f"' "$TERMLOG" >"$TMP/term_f.jsonl" + : >"$TMP/term_g.jsonl"; grep -F '"t-g"' "$TERMLOG" >"$TMP/term_g.jsonl" + python3 "$REPORT" "$TMP/term_f.jsonl" 2>&1 | tr -s ' ' | grep -Fqx " --for 1 100.0% (none)" \ + && python3 "$REPORT" "$TMP/term_g.jsonl" 2>&1 | tr -s ' ' | grep -Fqx " --for 1 0.0% grep (1)" \ + && ok "T10 terminality: the k=5 window edge holds from both sides (5 calls out, 4 calls in)" \ + || no "T10 terminality: the window edge is off by one — see $TMP/term_f.jsonl / $TMP/term_g.jsonl" + python3 "$REPORT" "$TERMLOG" >"$TMP/term2.out" 2>&1 + cmp -s "$TMP/term.out" "$TMP/term2.out" \ + && ok "T11 terminality: two runs over the same log are byte-identical (ties broken by name)" \ + || no "T11 terminality: the report is not deterministic — diff $TMP/term.out $TMP/term2.out" + # T12: the 200-character `detail` cap, which a REAL log hit on its first reading. A line cut + # mid-flag must not be filed under the prefix that survived — `--qualit` counted apart from + # `--quality-delta` splits one verb's n across two rows and understates both. Cut before any flag + # is a different unknown again: the verb may be one character past the cap. + TRUNCLOG="$TMP/term_trunc.jsonl"; : >"$TRUNCLOG" + TERMLOG_MAIN="$TERMLOG"; TERMLOG="$TRUNCLOG" + THEAD='cd /x/' + TCUT=0 + for TTAIL in ' && ./build/ripwire . --qualit' ' && ./build/ripwire /x/repo/src'; do + TCUT=$(( TCUT + 1 )) + # exactly 200 characters: the cap, reached mid-flag on the first line and mid-path on the second + TPAD="$( printf '%*s' "$(( 200 - ${#THEAD} - ${#TTAIL} ))" '' | tr ' ' 'a' )" + termrip 1 "t-cut$TCUT" "${THEAD}${TPAD}${TTAIL}" + done + TERMLOG="$TERMLOG_MAIN" + python3 "$REPORT" "$TRUNCLOG" 2>&1 | tr -s ' ' >"$TMP/trunc.sq" + grep -Fqx " --qualit... 1 100.0% (none)" "$TMP/trunc.sq" \ + && grep -Fqx " (truncated) 1 100.0% (none)" "$TMP/trunc.sq" \ + && ok "T12 terminality: a detail cut at the 200-char cap is labelled truncated, never filed under the surviving prefix" \ + || no "T12 terminality: the truncation labels are wrong — see $TRUNCLOG and $TMP/trunc.sq" else no "M26 meter: bench/substitution_report.py does not exist" no "M27b meter: bench/substitution_report.py does not exist" + no "T1 terminality: bench/substitution_report.py does not exist" fi # ═══════════════════════════════════════════════════════════════════════════════════════════════════