Skip to content

test(cli): bind AGENTS.md claims to the CLI's real argument parsers - #81

Merged
suhaanthayyil merged 3 commits into
entireio:mainfrom
mvanhorn:feat/agents-md-flag-registry-parity
Aug 7, 2026
Merged

test(cli): bind AGENTS.md claims to the CLI's real argument parsers#81
suhaanthayyil merged 3 commits into
entireio:mainfrom
mvanhorn:feat/agents-md-flag-registry-parity

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Why this matters

AGENTS.md is a shipped artifact, not documentation. init-agents upserts a pointer to it into
every consuming project and agent-guide prints it, so when it drifts from the code, agents in
other people's repositories act on stale rules. Nothing currently binds it to the CLI it
describes.

Three drifts were live on main at f6783ca:

Where Guide says Code does
AGENTS.md:72 edges has "no --to/--from/--relation filter" parseProviderFlags accepts all three (root.go:519,525,531)
help.go:237 edges flags: bare providerFlagDocs same three flags are real, so edges --help never listed them
help.go:192 impact --depth documented as 1 impact.go:164 sets Depth: 2

The first one has a cost: agents are told to stream the whole graph and grep client-side when a
server-side filter exists. On a 1,234-file repo that is 187,770 relation records instead of a
bounded query, and AGENTS.md:35 separately warns "Never edges for this (full stream)", advice
the missing-filter claim makes unavoidable.

What this adds

TestAgentGuideMatchesFlagRegistry extracts the closed set of checkable claims from AGENTS.md
and asserts them against the real argument parsers, not against another hand-maintained list:

  • every entire graph <command> named resolves in both commandDocs and the Run switch;
  • every --flag attributed to a command is accepted by that command's actual parser
    (parseProviderFlags, parseImpactFlags, parseNeighborFlags, and so on);
  • every explicit negative claim ("there is no --x filter") fails if the parser accepts it;
  • every documented (default: X) matches both the help registry and the parser's real default.

Failures name which of the three sources is wrong, so the fix is unambiguous:

AGENTS.md:72 guide/parser drift: guide says edges has no --to filter,
  but its real argument parser accepts it; fix AGENTS.md
AGENTS.md:72 parser/help drift: edges parser accepts --to,
  but internal/cli/help.go does not document it; fix help.go

Binding to the parser rather than to flagDoc is the load-bearing choice. An earlier draft
compared AGENTS.md to commandDocs and passed on the broken tree. Both were wrong in the
same direction, so doc-to-doc parity was satisfied while both disagreed with the parser. That is
the same argument as #76: derive from behavior, not from hand-kept lists.

Widening attribution to per-command bullet lists (where most of the guide's flags actually live)
then surfaced a third drift on its own: neighbors and impact both accept --profile
(neighbors.go:233, impact.go:214, default full) and AGENTS.md documents it, but
commandDocs omitted it.

Scope

Deliberately narrow. The test checks extractable claims only: commands named, flags in code
blocks and per-command bullets, explicit negative claims, and documented defaults. Prose,
examples, and doctrine are untouched, so rewording a paragraph does not break CI (proven below).
Where a command's parser is not reachable from the test (commit, diff, checkpoint,
capabilities, doctor, version) the claim is skipped with a logged reason rather than passing
silently.

I did not change impact --depth at runtime; the doc moved to match the code. If the intent was
a default of 1, that is a one-line change in impact.go:164 instead, and I am happy to flip it.

Verification

the three drifts, the failing test, and the pass

Fail-without / pass-with, both captured:

$ git checkout upstream/main -- AGENTS.md internal/cli/help.go
$ go test ./internal/cli -run TestAgentGuideMatchesFlagRegistry -count=1
--- FAIL: TestAgentGuideMatchesFlagRegistry (0.00s)
    AGENTS.md:46 parser/help drift: neighbors parser accepts --profile, but internal/cli/help.go does not document it
    AGENTS.md:58 parser/help drift: impact parser accepts --profile, but internal/cli/help.go does not document it
    AGENTS.md:72 guide/parser drift: guide says edges has no --to filter, but its real argument parser accepts it
    ... (11 failures across all three drifts)
FAIL

$ # with this PR
$ go test ./internal/cli -run TestAgentGuideMatchesFlagRegistry -count=1
ok      github.com/entireio/entire-graph/internal/cli    0.675s

Coverage probe: corrupting one prose-only flag claim fails the test:

$ # AGENTS.md:45, --internal-only -> --totally-bogus-flag
--- FAIL: AGENTS.md:45 guide/parser drift: neighbors uses --totally-bogus-flag,
      but its real argument parser does not accept it

Rewording an ordinary When: sentence still passes. Full local gate: gofmt -l -s clean,
go vet ./... clean, go test ./... ok, go test -race ./internal/cli ok, statusline suite
151 passed / 0 failed.

internal/cli/agentguide_test.go also unit-tests its own extractor and parser probe
(TestParseGuideNegativeClaims, TestParseGuideDefaultClaims, TestParseGuideProseFlagClaims,
TestParserAcceptsFlag), including the not-reachable case, so the checker itself is not taken on
faith.

No new dependencies. No schema change. No runtime behavior change.

Note on #80: it also touches AGENTS.md, but only appends the init-agents pointer block at the
end of the file, so these should not conflict.

AI was used for assistance.

…ations

Bullet lists under each command heading are where most of AGENTS.md documents
flags, and they carry no literal `entire graph <command>` for the parser to
attribute. Fall back to the enclosing section heading so those claims are
checked too; explicit attribution still wins when a span names a command.

Widening the check surfaced a third drift: the neighbors and impact parsers
both accept --profile (neighbors.go:233, impact.go:214, default full), and
AGENTS.md documents it, but commandDocs omitted it, so --help never showed it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a contract test that validates extractable claims in the shipped AGENTS.md guide against the CLI’s real argument parsers (and cross-checks against internal/cli/help.go), then updates the guide and help registry to eliminate known drifts (notably: edges filtering flags, and impact --depth default).

Changes:

  • Document --profile for neighbors and impact, and document edges’ server-side filters (--to/--from/--relation) in internal/cli/help.go.
  • Add internal/cli/agentguide_test.go to parse claims from AGENTS.md and assert them against the actual flag parsers and help registry.
  • Update AGENTS.md to reflect edges filtering support and the correct default for impact --depth.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
internal/cli/help.go Updates CLI help registry to accurately document --profile and edges filtering flags.
internal/cli/agentguide_test.go Adds contract tests binding AGENTS.md claims to real CLI parsers and help docs.
AGENTS.md Updates guide text/usage to match the actual CLI behavior for edges filters and impact default depth.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread internal/cli/agentguide_test.go
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

@suhaanthayyil suhaanthayyil left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed independently. Approving.

Zero runtime behaviour change — 3 files: AGENTS.md (docs), internal/cli/help.go (the commandDocs table), and a new internal/cli/agentguide_test.go. No schema, no ADR, no internal/sem.

The test is real. Mutation-verified in a scratch clone: injecting entire graph explain --repo . into AGENTS.md fails with agentguide_test.go:48: AGENTS.md:24 names command "explain", but commandDocs/Run do not both register it; injecting --totally-bogus-flag fails at :56.

Why it matters here specifically. Docs-to-CLI drift is not cosmetic in this repo — it is a silent-failure class. An agent that runs a documented command, gets a non-zero exit, and falls back to grep produces a benchmark cell that looks clean while the tool was never reached. This PR binds the claims to the real parsers, and it caught three live drifts on main.

One scope limitation worth recording (not blocking): the guarded artifact is repo-root AGENTS.md, but the one that actually ships is internal/cli/agents.go const agentGuide — printed by agent-guide and written into consumers' repos by init-agents. That string stays unguarded. Worth a follow-up so the shipped guide gets the same binding.

@mvanhorn

mvanhorn commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @suhaanthayyil for the independent verification pass and the merge. That review rigor is appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants