Skip to content

fix: take the shell away from the claude reviewer - #123

Merged
kim-em merged 1 commit into
mainfrom
reviewer-no-shell
Aug 21, 2026
Merged

fix: take the shell away from the claude reviewer#123
kim-em merged 1 commit into
mainfrom
reviewer-no-shell

Conversation

@kim-em

@kim-em kim-em commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

This PR takes the shell away from the claude reviewer, which it turned out to have all along.

run_claude passed --allowedTools Read Grep Glob. That flag grants permission within whatever tool set exists; it does not restrict the set. The reviewer therefore still had Bash, and used it heavily. The tool trace added in #118 shows it plainly, across 20 rubric runs:

tool        ok   failed/denied
Bash       278              17
Grep        80               1
Read        49               0
WebFetch     0               1

Sixty-nine percent of its tool calls were Bash, and 278 of them succeeded. Reproduced directly, in the environment shape reviewer_env builds: claude -p --disable-slash-commands --allowedTools Read Grep Glob answers SHELL_IS_AVAILABLE.

The rest of this module is built on that not being the case. reviewer_env calls the isolation load-bearing "with public transcripts and no redaction gate, a prompt-injected reviewer must have nothing worth leaking", and names its residual as a reviewer reading its own key from /proc/self/environ, something that "needs a proxy or uid-separation to close". A shell, plus the network egress a reviewer needs to reach its provider, turns that residual into a direct exfiltration path rather than a documented limitation. run_pi states the intended property outright for the OpenRouter reviewers: "a read-only tool set (PI_TOOLS, no bash) means it has no shell to leak it with."

--tools restricts the built-in set, so both flags are now passed: one removes the tool, the other keeps the remaining three from prompting a headless round to a halt. --disallowedTools was the other candidate and is worse, since it blocks Bash but leaves Write and Edit in the set.

Verified end to end that a review still works on the restricted set: a realistic reuse question resolved in seven turns using Grep, Glob and Read, correctly.

tests/test_reviewer_no_shell.py asserts the resulting tool SET rather than a flag's spelling, so a later rewrite that keeps the reviewer read-only passes, and one that quietly readmits a shell does not.

🤖 Prepared with Claude Code

run_claude passed --allowedTools Read Grep Glob. That grants permission within
whatever tool set exists; it does not restrict the set. The reviewer still had
Bash and used it: of 427 traced tool calls across 20 rubric runs, 295 were Bash
and 278 succeeded. Reproduced directly in the shape reviewer_env builds, where
'claude -p --disable-slash-commands --allowedTools Read Grep Glob' answers
SHELL_IS_AVAILABLE.

The rest of this module is built on that not being so. reviewer_env calls the
isolation load-bearing 'with public transcripts and no redaction gate', and
names its residual as a reviewer reading its own key from /proc/self/environ; a
shell plus the egress a reviewer needs to reach its provider turns that residual
into a direct exfiltration path. run_pi already states the intended property for
the OpenRouter reviewers: 'a read-only tool set (PI_TOOLS, no bash) means it has
no shell to leak it with.'

Pass --tools as well, which restricts the built-in set. --disallowedTools was
the other candidate and is worse: it blocks Bash but leaves Write and Edit.
Verified end to end that a reuse review still works on the restricted set, in
seven turns using Grep, Glob and Read.

The test asserts the SET, not a flag's spelling, so a rewrite that keeps the
tools read-only passes and one that quietly readmits a shell does not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T1zpkPwUh41oU3mtSQGDVU
@kim-em
kim-em requested a review from a team as a code owner August 21, 2026 01:51
@kim-em
kim-em merged commit f854a7e into main Aug 21, 2026
1 check passed
@kim-em
kim-em deleted the reviewer-no-shell branch August 21, 2026 04:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant