docs: add versioned CLI capture matrix contract - #257
harshitethic wants to merge 3 commits into
Conversation
| "capture_method": "cli_hooks", | ||
| "setup_command": "agent-strace setup --cli claude", | ||
| "config_location": "~/.claude/settings.json", | ||
| "rollback": "Remove the AgentTrace hook commands written by agent-strace setup from ~/.claude/settings.json.", |
There was a problem hiding this comment.
agent-strace setup --cli claude does not write this file. cmd_setup prints JSON and tells the user to add it manually. This rollback text incorrectly implies that setup modified ~/.claude/settings.json. Describe removal of the manually added hook block, or make setup install it before documenting this behavior.
| with self.subTest(provider=provider["id"]): | ||
| self.assertIn(provider["name"], self.docs) | ||
| self.assertIn(provider["setup_command"], self.docs) | ||
| self.assertIn(provider["config_location"], self.docs) |
There was a problem hiding this comment.
This only checks three identifying strings. Coverage and verification can contradict capture-matrix.json while the test still passes, despite the human and machine-readable consistency goal. Assert the rendered coverage and status values, or generate the Markdown table from the JSON.
|
|
||
| def test_verified_entries_require_reproducibility_metadata(self): | ||
| for provider in self.matrix["providers"]: | ||
| if provider["verification_status"] != "verified": |
There was a problem hiding this comment.
verification_status is never validated. A typo such as verifiedd bypasses the reproducibility checks and still passes the contract test. Add an allowed-status assertion, then require the metadata for the exact verified value.
| "file": "provider_defined", | ||
| "command": "provider_defined", | ||
| "error": "provider_defined", | ||
| "stop": "unavailable" |
There was a problem hiding this comment.
The generated Codex config registers a Stop hook, and test_codex_hooks.py verifies that handle_stop records the assistant response. Marking stop as unavailable contradicts the shipped adapter. Keep session_end unavailable, but mark stop captured or rename this capability if it means something else.
| "capture_method": "cli_hooks", | ||
| "setup_command": "agent-strace setup --cli codex", | ||
| "config_location": "~/.codex/hooks.json", | ||
| "rollback": "Remove the AgentTrace hook commands written by agent-strace setup from ~/.codex/hooks.json.", |
There was a problem hiding this comment.
setup --cli codex replaces the entire hooks.json with write_text(). Removing the AgentTrace commands cannot restore hooks that were overwritten, so this rollback is not reversible for existing users. Merge with the existing config or create a backup, then document the actual restoration step.
This branch has not been deployed
2 participants
Summary
Adds the first machine-readable + human-readable capture matrix for the CLI hook adapters, scoped to Claude Code and OpenAI Codex as requested in #239.
What changes
docs/capture-matrix.jsondefines provider setup/config/rollback, privacy notes, known gaps, and per-capability coverage.docs/capture-matrix.mdgives the reviewer-facing version of the same boundaries.tests/test_capture_matrix.pykeeps provider IDs unique, enforces the coverage schema, checks the human docs mention every machine-readable provider, and prevents an entry from claimingverifiedwithout a tested version, verification date, and fixture/test reference.Trust boundary
The current repository documents adapter behavior, but it does not contain a clean-environment verification record with exact Claude/Codex versions. This PR therefore marks both entries
unverifiedrather than fabricating version/date evidence. The structure makes that missing evidence visible and gives a deterministic place for #239's future verification command to update.For Codex, the matrix intentionally does not claim session-end/stop coverage because the current integration docs only claim session start, prompts, responses, and PreToolUse/PostToolUse activity.
Scope
This is a focused foundation for #239, not a claim to close the full issue. The remaining work is the executable setup/doctor verification path and automated stale-entry checks tied to real provider-version fixtures.
Validation
The added test is dependency-free (
unittest+ stdlib JSON/pathlib). I could not run the local suite because the connected development machine is offline; upstream CI is the verification gate for this branch.Refs #239.