Skip to content

feat(review,audit): deterministic coverage and cited-line verification - #643

Open
iagogfe wants to merge 1 commit into
DietrichGebert:mainfrom
iagogfe:feat/deterministic-coverage
Open

feat(review,audit): deterministic coverage and cited-line verification#643
iagogfe wants to merge 1 commit into
DietrichGebert:mainfrom
iagogfe:feat/deterministic-coverage

Conversation

@iagogfe

@iagogfe iagogfe commented Jul 27, 2026

Copy link
Copy Markdown

What

Two additions to ponytail-review and ponytail-audit, both mechanical, zero new scope:

  1. Deterministic coverage. The file list comes from git diff --stat (review) or git ls-files (audit), and every file ends in one of two states: reviewed, or skipped with a reason. Reports close with covered: N/M. Lazy means reading less per file, never silently reading fewer files.
  2. Cited-line verification. Every path:line in a finding is checked with sed -n '<line>p' before it ships. A citation pointing at } is a finding the author never locates.

Why

On larger diffs the model quietly cuts corners: it picks the "interesting" files and the report reads as if everything was covered. And line references drift. Both are shape problems, not judgment problems, so the fix is a contract on the output, not more prose asking the model to be careful.

Measured

Same model, same 12-file Rust diff, same prompt, one run each way:

without with
coverage varied between runs, no declaration 12/12, per-file status
cited lines 1 of 6 pointed at a closing brace 3/4 exact, 1 in the right block
cost 86k tokens, 17 tool calls 75k tokens, 9 tool calls

The cost drop was unexpected but repeatable in my setup: enumerating first replaces exploratory wandering.

n=1 per arm, so take the numbers as direction, not proof. The shape guarantee (coverage table always present) is the part that held on every run I did.

Checks

  • .openclaw/skills/ mirrors regenerated with scripts/build-openclaw-skills.js (the sync test caught my first push missing them).
  • Local suite: 81/82 pass. The one failure (csv: correct pandas one-liner) reproduces on pristine main in my environment (no pandas installed), so it's environmental, not from this change.

Fit

Kept to the house style: recipe form, no prohibitions, a few lines per skill. net: stays the closing metric; covered: precedes it. No new dependencies.

Two failure modes surface on larger diffs: the model silently reviews
fewer files than the diff contains, and cited line numbers drift off
target. Both are fixed with mechanics, not judgment: the file list comes
from git, every file ends reviewed or skipped-with-reason, and each
cited line is checked with sed before it ships. Reports close with a
covered: N/M line so coverage is a checkable fact instead of a feeling.

Measured on a 12-file Rust diff (same model, same prompt): without the
contract, coverage varied between runs and one citation in six pointed
at a closing brace; with it, 12/12 files with per-file status and the
review used fewer tool calls (9 vs 17) and fewer tokens (75k vs 86k).
@iagogfe
iagogfe force-pushed the feat/deterministic-coverage branch from 0c5b877 to e41ddb3 Compare July 28, 2026 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants