feat(review,audit): deterministic coverage and cited-line verification - #643
Open
iagogfe wants to merge 1 commit into
Open
feat(review,audit): deterministic coverage and cited-line verification#643iagogfe wants to merge 1 commit into
iagogfe wants to merge 1 commit into
Conversation
Two failure modes surface on larger diffs: the model silently reviews fewer files than the diff contains, and cited line numbers drift off target. Both are fixed with mechanics, not judgment: the file list comes from git, every file ends reviewed or skipped-with-reason, and each cited line is checked with sed before it ships. Reports close with a covered: N/M line so coverage is a checkable fact instead of a feeling. Measured on a 12-file Rust diff (same model, same prompt): without the contract, coverage varied between runs and one citation in six pointed at a closing brace; with it, 12/12 files with per-file status and the review used fewer tool calls (9 vs 17) and fewer tokens (75k vs 86k).
iagogfe
force-pushed
the
feat/deterministic-coverage
branch
from
July 28, 2026 01:04
0c5b877 to
e41ddb3
Compare
aakashdev26
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two additions to
ponytail-reviewandponytail-audit, both mechanical, zero new scope:git diff --stat(review) orgit ls-files(audit), and every file ends in one of two states: reviewed, or skipped with a reason. Reports close withcovered: N/M. Lazy means reading less per file, never silently reading fewer files.path:linein a finding is checked withsed -n '<line>p'before it ships. A citation pointing at}is a finding the author never locates.Why
On larger diffs the model quietly cuts corners: it picks the "interesting" files and the report reads as if everything was covered. And line references drift. Both are shape problems, not judgment problems, so the fix is a contract on the output, not more prose asking the model to be careful.
Measured
Same model, same 12-file Rust diff, same prompt, one run each way:
The cost drop was unexpected but repeatable in my setup: enumerating first replaces exploratory wandering.
n=1 per arm, so take the numbers as direction, not proof. The shape guarantee (coverage table always present) is the part that held on every run I did.
Checks
.openclaw/skills/mirrors regenerated withscripts/build-openclaw-skills.js(the sync test caught my first push missing them).csv: correct pandas one-liner) reproduces on pristine main in my environment (no pandas installed), so it's environmental, not from this change.Fit
Kept to the house style: recipe form, no prohibitions, a few lines per skill.
net:stays the closing metric;covered:precedes it. No new dependencies.