feat: record which tools a claude reviewer actually used - #118
Merged
Conversation
The engine kept only the final answer. `--output-format json` returns one document whose `result` is the review, and the session file lands in the throwaway HOME that cleanup_rev_home deletes, so nothing said whether a finding came from reading the code or from the model's recollection of it. 'Verify before you assert: name the declaration and show the grep hit' is the central instruction of rubrics/_common.md, and it was unfalsifiable. Ask for stream-json instead. The terminal result event carries every field the json format did, and the events before it give a compact ordered trace of each tool call and the path or pattern it was about. Never what a call returned: the store and the archive are both public, and file contents there would republish the PR under review. Bounded at 40 calls, arguments clipped, and parsing is tolerant of interleaved noise, a malformed line, or a stream that never produced its terminal event. Claude only for now; codex parses a different event stream and keeps no trace either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T1zpkPwUh41oU3mtSQGDVU
kim-em
marked this pull request as draft
August 14, 2026 07:47
The first version recorded the model's own words. Grep patterns and Bash commands are chosen by a model reading an untrusted diff, and both persisted sinks are public, so a prompt-injected reviewer could read its credential from /proc/self/environ (which reviewer_env already concedes it can) and launder it out through the next tool argument. --allowedTools governs permission, not visibility, so even a denied Bash request arrived carrying its command. Record a path, and only after resolving it inside the workspace and finding it already exists. A read-only reviewer cannot create the file whose name would carry a secret, and everything already there is public; anything else becomes a bucket, since 'it tried to read outside the workspace' is what an audit wants to see. Each entry now carries its paired tool_result outcome, because a request is not an inspection and a denied Read must not read as a successful one. On the parse-error path the raw stream was persisted, and under stream-json its tail contains tool_result bodies. raw_stdout joins PRIVATE_KEYS: kept in process for a local operator, never written to a public sink. Also require that the CLI did not report failure before accepting a verdict, so a partial or injected result carrying a well-formed marker cannot be published as one, and record api_error_status, which is the field that names an API failure arriving as is_error=true with subtype=success. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T1zpkPwUh41oU3mtSQGDVU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR records which tools a claude reviewer actually used, alongside what it concluded.
The engine kept only the final answer.
--output-format jsonreturns one document whoseresultis the review, and the reviewer's session file lands in the throwaway HOME thatcleanup_rev_homedeletes, so nothing anywhere said whether a finding came from reading the code or from the model's recollection of it. "Verify before you assert: name the declaration and show thegrephit" is the central instruction ofrubrics/_common.md, and it was unfalsifiable.Asking for
stream-jsonfixes that without changing the review. The terminalresultevent carries every field the json format supplied, and the events before it give a compact ordered trace of each tool call and the path or pattern it was about.Never what a call returned. The store is a checkout of the public
reviewsbranch and the archive lands in TauCetiData, so file contents in either would republish the PR under review. The trace is bounded at 40 calls with arguments clipped, so one grep-heavy rubric cannot bloat a persisted record. Parsing is tolerant of interleaved non-JSON and a malformed line, and a stream that never produced its terminal event takes the same path a malformed json document took, now with a diagnosis rather than an empty string.Claude only for now. Codex parses a different event stream and keeps no trace either; that is worth a follow-up.
🤖 Prepared with Claude Code