test(cli): bind AGENTS.md claims to the CLI's real argument parsers - #81
Conversation
…ations Bullet lists under each command heading are where most of AGENTS.md documents flags, and they carry no literal `entire graph <command>` for the parser to attribute. Fall back to the enclosing section heading so those claims are checked too; explicit attribution still wins when a span names a command. Widening the check surfaced a third drift: the neighbors and impact parsers both accept --profile (neighbors.go:233, impact.go:214, default full), and AGENTS.md documents it, but commandDocs omitted it, so --help never showed it.
There was a problem hiding this comment.
Pull request overview
This PR adds a contract test that validates extractable claims in the shipped AGENTS.md guide against the CLI’s real argument parsers (and cross-checks against internal/cli/help.go), then updates the guide and help registry to eliminate known drifts (notably: edges filtering flags, and impact --depth default).
Changes:
- Document
--profileforneighborsandimpact, and documentedges’ server-side filters (--to/--from/--relation) ininternal/cli/help.go. - Add
internal/cli/agentguide_test.goto parse claims fromAGENTS.mdand assert them against the actual flag parsers and help registry. - Update
AGENTS.mdto reflectedgesfiltering support and the correct default forimpact --depth.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| internal/cli/help.go | Updates CLI help registry to accurately document --profile and edges filtering flags. |
| internal/cli/agentguide_test.go | Adds contract tests binding AGENTS.md claims to real CLI parsers and help docs. |
| AGENTS.md | Updates guide text/usage to match the actual CLI behavior for edges filters and impact default depth. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
suhaanthayyil
left a comment
There was a problem hiding this comment.
Reviewed independently. Approving.
Zero runtime behaviour change — 3 files: AGENTS.md (docs), internal/cli/help.go (the commandDocs table), and a new internal/cli/agentguide_test.go. No schema, no ADR, no internal/sem.
The test is real. Mutation-verified in a scratch clone: injecting entire graph explain --repo . into AGENTS.md fails with agentguide_test.go:48: AGENTS.md:24 names command "explain", but commandDocs/Run do not both register it; injecting --totally-bogus-flag fails at :56.
Why it matters here specifically. Docs-to-CLI drift is not cosmetic in this repo — it is a silent-failure class. An agent that runs a documented command, gets a non-zero exit, and falls back to grep produces a benchmark cell that looks clean while the tool was never reached. This PR binds the claims to the real parsers, and it caught three live drifts on main.
One scope limitation worth recording (not blocking): the guarded artifact is repo-root AGENTS.md, but the one that actually ships is internal/cli/agents.go const agentGuide — printed by agent-guide and written into consumers' repos by init-agents. That string stays unguarded. Worth a follow-up so the shipped guide gets the same binding.
|
Thanks @suhaanthayyil for the independent verification pass and the merge. That review rigor is appreciated. |
Why this matters
AGENTS.mdis a shipped artifact, not documentation.init-agentsupserts a pointer to it intoevery consuming project and
agent-guideprints it, so when it drifts from the code, agents inother people's repositories act on stale rules. Nothing currently binds it to the CLI it
describes.
Three drifts were live on
mainat f6783ca:AGENTS.md:72edgeshas "no--to/--from/--relationfilter"parseProviderFlagsaccepts all three (root.go:519,525,531)help.go:237edgesflags: bareproviderFlagDocsedges --helpnever listed themhelp.go:192--depthdocumented as1impact.go:164setsDepth: 2The first one has a cost: agents are told to stream the whole graph and grep client-side when a
server-side filter exists. On a 1,234-file repo that is 187,770 relation records instead of a
bounded query, and
AGENTS.md:35separately warns "Neveredgesfor this (full stream)", advicethe missing-filter claim makes unavoidable.
What this adds
TestAgentGuideMatchesFlagRegistryextracts the closed set of checkable claims fromAGENTS.mdand asserts them against the real argument parsers, not against another hand-maintained list:
entire graph <command>named resolves in bothcommandDocsand theRunswitch;--flagattributed to a command is accepted by that command's actual parser(
parseProviderFlags,parseImpactFlags,parseNeighborFlags, and so on);--xfilter") fails if the parser accepts it;(default: X)matches both the help registry and the parser's real default.Failures name which of the three sources is wrong, so the fix is unambiguous:
Binding to the parser rather than to
flagDocis the load-bearing choice. An earlier draftcompared
AGENTS.mdtocommandDocsand passed on the broken tree. Both were wrong in thesame direction, so doc-to-doc parity was satisfied while both disagreed with the parser. That is
the same argument as #76: derive from behavior, not from hand-kept lists.
Widening attribution to per-command bullet lists (where most of the guide's flags actually live)
then surfaced a third drift on its own:
neighborsandimpactboth accept--profile(
neighbors.go:233,impact.go:214, defaultfull) andAGENTS.mddocuments it, butcommandDocsomitted it.Scope
Deliberately narrow. The test checks extractable claims only: commands named, flags in code
blocks and per-command bullets, explicit negative claims, and documented defaults. Prose,
examples, and doctrine are untouched, so rewording a paragraph does not break CI (proven below).
Where a command's parser is not reachable from the test (
commit,diff,checkpoint,capabilities,doctor,version) the claim is skipped with a logged reason rather than passingsilently.
I did not change
impact --depthat runtime; the doc moved to match the code. If the intent wasa default of 1, that is a one-line change in
impact.go:164instead, and I am happy to flip it.Verification
Fail-without / pass-with, both captured:
Coverage probe: corrupting one prose-only flag claim fails the test:
Rewording an ordinary
When:sentence still passes. Full local gate:gofmt -l -sclean,go vet ./...clean,go test ./...ok,go test -race ./internal/cliok, statusline suite151 passed / 0 failed.
internal/cli/agentguide_test.goalso unit-tests its own extractor and parser probe(
TestParseGuideNegativeClaims,TestParseGuideDefaultClaims,TestParseGuideProseFlagClaims,TestParserAcceptsFlag), including the not-reachable case, so the checker itself is not taken onfaith.
No new dependencies. No schema change. No runtime behavior change.
Note on #80: it also touches
AGENTS.md, but only appends theinit-agentspointer block at theend of the file, so these should not conflict.
AI was used for assistance.