Skip to content

An agent judges only on an argv someone has actually run - #41

Merged
bharathm03 merged 2 commits into
developmentfrom
fix/headless-verified-argvs
Aug 28, 2026
Merged

An agent judges only on an argv someone has actually run#41
bharathm03 merged 2 commits into
developmentfrom
fix/headless-verified-argvs

Conversation

@bharathm03

Copy link
Copy Markdown
Contributor

Follow-up to #38, from digging into the three items left open there. Two of the review's claims did not survive contact; the third got sharper.

1. Unverified argvs armed the judge

cursor-agent and mistral-vibe declared headless entries that had never been executed — both CLIs refuse every invocation on this machine without CURSOR_API_KEY / MISTRAL_API_KEY, so their reach came from reading help output. AgentSpec.headless says outright that an entry goes in only after running it, and the cost of breaking that is not a mislabel: any non-sealed reach makes an agent judgeCapable, arming a supervisor over the user's working tree on an argv nobody has run.

Both entries are removed. Naming is unaffected — a need: "none" call borrows an installed agent.

Copilot's readonly entry stays, against the review's suggestion. Writes were re-tested through three vectors — a plain file create, an explicit PowerShell shell call, and git init's side effect — and the system blocked all three ("The system has blocked this operation", "insufficient permissions"), with a read confirmed working. That is what "the argv provably restricts the tool to reads" means.

2. A failed spawn cost the session its name

The single "already attempted" flag answered three questions with one bit and got two wrong: a signed-out CLI or a timeout marked the conversation spent, so the session could never be named afterwards — not even once the cause was fixed. Verified against the real CLIs: a signed-out cursor-agent and vibe both exit 1, which is exactly the path that used to burn the attempt.

TitleAttempts (new, unit-tested) splits the three states a count cannot carry at once:

  • inFlight — mutual exclusion, released in a finally however the spawn ends
  • failures — bounded retry (2), because each attempt costs a 45s budget
  • done — named, or given up on

generateTitleFromContext now returns why it failed rather than a bare null, so "nothing installed can serve this" ends the budget outright while a spawn that merely failed does not.

3. Scratch state accumulated in a directory nothing reclaims

Two of the review's claims here were wrong: the directory is created (copilot mkdirs its own home), and concurrent spawns sharing one home do not contend — two ran together and each wrote its own session.

The real defect is growth. COPILOT_HOME pointed at one fixed directory per tool, which kept every session it was ever handed: ~51KB apiece plus a 352KB uncheckpointed WAL after two calls, in a %TEMP% Windows does not reclaim. HeadlessCommand.scratchEnv now names the vars and the runner creates the directory per spawn and deletes it with the spawn. Verified end to end against the real copilot CLI: authenticates from a fresh empty home, returns a title in ~15s, leaves nothing behind.

Severity note: this was a disk leak, not a correctness bug — the growth was always in the scratch dir, so no naming run ever reached the user's real history.

Testing

bun run --filter antgrid-bridge test — 2984 pass, 0 fail. One unidentified flake in the first of three full runs, not reproduced in the two after it; the title/headless suites are green in isolation. New coverage: tests/title-attempts.test.ts (exclusion, budget, the flat-key prefix trap) and a scratchEnv block in tests/headless.test.ts (dir is real during the spawn, gone after, distinct per spawn, untouched when unasked).

cursor-agent and mistral-vibe declared headless argvs that had never been executed — both CLIs refuse every invocation on this machine without CURSOR_API_KEY / MISTRAL_API_KEY, so their reach was inferred from help output rather than measured. AgentSpec.headless says outright that an entry goes in only after running it, and the cost of breaking that is not a mislabel: any non-sealed reach makes the agent judgeCapable, arming a supervisor over the user's working tree on an argv nobody has run. Both entries are gone; naming still works by borrowing an installed agent. Copilot's readonly entry stays — writes were re-tested through three vectors (plain create, an explicit shell call, and git init's side effect) and the system blocked all three.

A failed naming spawn no longer costs the session its name. The single 'attempted' flag answered three questions with one bit and got two wrong: a signed-out CLI or a timeout marked the conversation spent, so it could never be named even after the cause was fixed. TitleAttempts splits them — an in-flight claim released in a finally, a bounded failure count, and a terminal done — and generateTitleFromContext now returns why it failed, so 'nothing installed can serve this' ends the budget while a spawn that merely failed does not.

State a CLI offers no switch to skip is redirected per spawn rather than to a fixed path. COPILOT_HOME pointed at one directory per tool, which kept every session it was ever handed: ~51KB apiece plus a 352KB uncheckpointed WAL after two calls, in a %TEMP% Windows does not reclaim. HeadlessCommand.scratchEnv names the vars, and the runner creates the directory and deletes it with the spawn.
Read against mistralai/mistral-vibe v2.24.5 after the entry was removed for being unrun. The removed argv was wrong in a sharper way than that: `ask` is the approval-gated profile ('Requires approval for tool executions'), while `plan` is the read-only one and the only builtin pinning write_file and edit to permission 'never'. Programmatic mode denies every callback it is handed, which is what made ask look read-only — a write fails closed on an approval nothing can answer.

That safety is config-level, never argv-level. An agent profile is only another config layer; ask contributes no bypass_tool_permissions key, so a user's own config survives it, and the loop returns EXECUTE before consulting any permission once that is set. Nothing raises an approval, so nothing is denied — and the same switch defeats `--agent plan`. Recorded on the entry so the argv is not reintroduced as 'readonly'.
@bharathm03

Copy link
Copy Markdown
Contributor Author

Source verification of the vibe entry

Checked out mistralai/mistral-vibe @ 50d99cf (v2.24.5, same version as the installed CLI) to verify the entry this PR removes. It was wrong in a sharper way than "never executed" — the removal is right, but the reasoning in the PR description understates it.

ask is not a read-only profile. Printed from the installed package, not just the clone:

ask          safety=neutral  desc='Requires approval for tool executions'
             overrides={'disabled_tools': ['exit_plan_mode']}
plan         safety=safe     desc='Read-only agent for exploration and planning'
             overrides={'tools': {'write_file': {'permission': 'never', ...},
                                  'edit': {'permission': 'never', ...}, ...}}

plan is the read-only one. ask restricts no write tool at all — its only override disables exit_plan_mode.

What made it look read-only is unrelated to the profile: cli/programmatic.py:161 answers every callback with session.deny_callback, which resolves to ApprovalDecisionType.DENY. So under default config a write does fail closed — on an approval that non-interactive mode refuses, exactly as the removed comment guessed, but for a reason that has nothing to do with --agent ask.

And that safety is config-level, which is the distinction HeadlessReach exists to draw. An agent profile is just another config layer (AgentProfileLayer, merged via apply_profile_overrides). bypass_tool_permissions is a top-level user-settable key, ask contributes no such key, so a user's ~/.vibe/config.toml value survives the profile. _loop.py:_should_execute_tool then short-circuits:

if self.bypass_tool_permissions:
    return ToolDecision(verdict=ToolExecutionResponse.EXECUTE, ...)

No approval is raised, so programmatic's blanket deny never fires. headless=True does not force enforcement either — bypass_approval = options.auto_approve or config.bypass_tool_permissions.

Consequences

  • The shipped entry was readonly on an argv that provably restricts nothing. On a machine with bypass_tool_permissions = true it was an unrestricted agent — and judgeCapable, so it could have been armed over the user's working tree.
  • The same switch defeats --agent plan, so even a corrected entry could only ever be transcript here — the same reach opencode and kilo carry, and for the same reason.
  • It remains unrun regardless (no MISTRAL_API_KEY), so no entry goes back in.

Recorded on the registry entry in 99ab8ba so the argv is not reintroduced as readonly.

@bharathm03
bharathm03 merged commit bd05f4d into development Aug 28, 2026
3 checks passed
@bharathm03
bharathm03 deleted the fix/headless-verified-argvs branch August 28, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant