T0: the report answers whether a verb ended the question - #34
Merged
Merged
Conversation
An output only saves tokens if it terminates the question that prompted it; one
that spawns the next command is net-additive. bench/substitution_report.py gains
§5, per-verb terminality, computed from the meter log it already reads: for each
ripwire call, look ahead inside its session -- to the next ripwire call, the
session end, or 5 calls, whichever comes first -- and ask whether a sweep-class
call followed. Verb, n, terminal%, and the first follow-up class that broke it.
Three definitional choices are the substance, and each is printed above the table
rather than left to the docs, because a terminality percentage read without its
window rule is a number somebody quotes wrong:
- the sweep set is WIDER than §1's `native` family. git diff/log/show --stat are
history retrieval, and a map followed by a raw git-history sweep did not
terminate the question any more than a map followed by grep did. They stay out
of the substitution ratio, where they are a different QUESTION rather than a
different tool for the same one, and they count here, where the question is
whether the answer landed;
- an EMPTY window counts terminal -- no sweep happened -- which is the
definition's softest spot, so the count of them is disclosed under the table
instead of folded in silently;
- n travels with every percentage, and a verb under n=10 carries an explicit
NOTE row. Not a significance test; a floor under the reader.
The verb itself is read lexically from `detail`: the ripwire word by basename
(behind `cd X && VAR=y`, at any path), then the first flag before any pipe or
redirect, so a pipeline's own flags are never mistaken for the verb. The skip
list of verb-agnostic options is deliberately NOT a mirror of the binary's ~70-row
dispatch table -- a mirror rots silently, and being wrong here costs one row shown
by name under a modifier rather than one hidden.
The 200-character `detail` cap is named in all three of its forms rather than
guessed at, because a real log hit two of them on the first reading: cut before
the ripwire word is `(unparsed)`, cut after it with no flag yet is `(truncated)`,
and cut mid-flag keeps the flag with a trailing `...`. Filing a half-written flag
under the prefix that survived would split one verb's n across two rows and
understate both.
Gate: test/hookcheck.sh arms T1-T12, on a synthetic fixture log whose answer is
known by construction -- a terminal --for, a non-terminal one followed by three
greps, windows truncated by the next ripwire call and by session end, the k=5 edge
asserted from BOTH sides (five non-sweep calls put a grep out of the window, four
leave it in), a git-history follow-up, an n<10 verb with its NOTE and an n>=10 verb
without one, an MCP row, a flagless map behind a cd prefix and ahead of a pipeline,
and both truncation labels. Exact table values, not "it did not crash": 11 of the
12 were red against the previous script. The gate's own sink cannot serve as the
fixture -- its contents change with every arm anyone adds above it.
Analysis-side only: no schema change, no hook change, and the levels stay in the
operator-local ledger. 397 gates green.
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe substitution report now extracts ripwire verbs and measures terminality within bounded, same-session follow-up windows. It reports per-verb and aggregate statistics. Documentation and hook checks cover parsing, boundaries, truncation, small samples, MCP rows, and deterministic output. ChangesTerminality analysis
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant substitution_report
participant session_order
participant window_verdict
participant terminality_row
substitution_report->>session_order: order rows by session and sequence
session_order->>window_verdict: inspect bounded follow-up calls
window_verdict-->>substitution_report: terminal status and sweep follow-up
substitution_report->>terminality_row: format per-verb statistics
terminality_row-->>substitution_report: terminality report row
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/SUBSTITUTION_METER.md`:
- Around line 492-502: Update the gate-coverage references in the documented
test/hookcheck.sh section from T1–T11 to T1–T12, and extend the synthetic
fixture description to mention T12’s detail-truncation label assertions.
Preserve the existing descriptions of T1–T11 and ensure both references are
updated consistently.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: d5326d60-3784-4c14-9989-0e0f95f9ae07
📒 Files selected for processing (3)
bench/substitution_report.pydocs/SUBSTITUTION_METER.mdtest/hookcheck.sh
Comment on lines
+492
to
+502
| `test/hookcheck.sh` section (11), arms M1–M27b and T1–T11: a row is written at the default global path with the | ||
| full field set; the rtk unwrap; the `unclassified` fallback; out-of-scope calls writing no row; the | ||
| `nudged`/`dedup`/`post_nudge`/`post_sweep`/`seq` fields; both arms; one global log across two repos; | ||
| and — the arm that matters most — an unwritable log costing the hooked command nothing. | ||
|
|
||
| Arms **T1–T11** pin §5 against a **synthetic fixture log** whose right answer is known by | ||
| construction — a terminal `--for`, a non-terminal one followed by three greps, windows truncated by | ||
| the next ripwire call and by session end, the k=5 window edge asserted from *both* sides (five | ||
| non-sweep calls put a grep out of the window, four leave it in), a git-history follow-up, an n<10 | ||
| verb with its NOTE row and an n≥10 verb without one, an MCP row, and a flagless map behind a `cd` | ||
| prefix and ahead of a pipeline carrying flags of its own. The assertions are exact table values, not |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Document T12 in the gate coverage.
Lines 492 and 497 state that coverage ends at T11. test/hookcheck.sh Lines 792-811 implements T12 for detail truncation labels. Update both references to T1–T12 and include truncation coverage in the fixture description.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/SUBSTITUTION_METER.md` around lines 492 - 502, Update the gate-coverage
references in the documented test/hookcheck.sh section from T1–T11 to T1–T12,
and extend the synthetic fixture description to mention T12’s detail-truncation
label assertions. Preserve the existing descriptions of T1–T11 and ensure both
references are updated consistently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Track T's instrument: §5 terminality-by-verb in the substitution report. Per ripwire call, a bounded look-ahead window (next ripwire call / session end / 5 calls) decides TERMINAL vs sweep-followed, with the first follow-up class recorded. Sweep set deliberately includes the raw git-history classes. Analysis-side only — schema, hook, and log untouched.
Honesty surfaces the definition forced: empty windows are counted and printed separately, never folded in; every percentage carries its n with an explicit small-n note; truncated
detailfields get three labeled forms instead of minting phantom verbs (the first draft split one verb's n across two rows on a 200-char cut — gated as arm T12).Red-first: 11 of 12 new hookcheck arms red against the pre-change script; 12/12 green after (151 hookcheck arms total). The k=5 window edge is gated from both sides. No new gate file; counts untouched. Full plain suite ALL PASS; quality-delta gating=0; determinism/xmllint clean. Mechanism documented in SUBSTITUTION_METER.md — no telemetry levels in any committed file.
🤖 Generated with Claude Code