Skip to content

fix(verify): agent-loop output budgets share the simple_text thinking-era default (4096 → 20000) - #382

Merged
gadievron merged 1 commit into
masterfrom
fix/issue290-verify-output-budget
Aug 28, 2026
Merged

fix(verify): agent-loop output budgets share the simple_text thinking-era default (4096 → 20000)#382
gadievron merged 1 commit into
masterfrom
fix/issue290-verify-output-budget

Conversation

@gadievron

Copy link
Copy Markdown
Collaborator

Summary

finding_verifier and agentic_enhancer/agent call binding.adapter.complete(...) directly (they need raw content blocks for the multi-turn tool conversation), so PR #242's raise of simple_text's default 8192→20000 never reached them: both loops kept a private MAX_TOKENS_PER_RESPONSE = 4096half the pre-#242 default — on the largest units in the scan. On the run that filed #290, 31 of 182 incomplete verifications self-declared "finish call truncated at max_tokens". A truncated finish is deliberately downgraded to verification-incomplete (FN-safe), so the undersized cap silently converted adjudications into needs_review.

Fix: extract llm.helpers.DEFAULT_MAX_TOKENS (20000, the #242 rationale comment moved with it); simple_text's signature default and both agent-loop constants now reference the same constant, so the two budget paths cannot drift apart again. Zero behavior change for simple_text callers; the verify/enhance per-turn cap rises 4096→20000 (a CAP, not a floor — models still stop at end_turn).

Note: this issue was sequenced behind #285/#286/#287 so a budget change is measurable (status honesty, error-vocabulary, fingerprint keying) — those landed in #376/#377/#378.

Scope notes (deliberate deviations from the issue's suggestions, with reasons):

  • suggestion 1's "read from configuration" and suggestion 3's PhaseBinding property are architecture/config-surface changes, not fixes — deferred; the shared constant delivers the anti-drift property they were after.
  • suggestion 4's full 16-site census guard is not added: the other 4096 sites (llm_reachability, report generator, context_enhancer, etc.) are single-shot simple_text-style calls with their own cost/shape tradeoffs this issue did not analyze; blanket-justifying them would be guessing. The drift test covers the agent loops the issue indicts.
  • verify appends raw tool results with no input cap, while enhance caps both and documents why #291 (the input cap on appended tool results) is the sibling gap, independent, and remains open.

The private-run figures (31/182) quoted from the issue are provenance-labeled there; the code defect and the fix contract are fully checkable and test-covered.

Test plan

3 hermetic tests: both agent-loop constants ARE the shared constant (identity, not coincidental equality); simple_text's signature default is that constant; no private numeric budget pins remain in the agent-loop sources.

Verification evidence (commands + results)
check command result
new + helper tests pytest tests/test_issue290_output_budget.py tests/test_llm_helpers.py tests/test_llm_helpers_unit.py -q 24 passed
mutation smoke fix reverted → new test file errors (import DEFAULT_MAX_TOKENS fails) — RED proof; re-applied → green
hermeticity oracle env -i HOME=/tmp/fakehome-noconfig python -m pytest tests/test_issue290_output_budget.py 3 passed
full suite pytest tests/ -q 3152 passed, 2 failed (pre-existing zig local-env, stash-verified), 32 skipped
lint ruff check . (CI scope) clean

Fixes #290

…-era default

finding_verifier and agentic_enhancer/agent call binding.adapter.complete
directly (they need raw content blocks for the multi-turn tool
conversation), so PR #242's raise of simple_text's default 8192→20000 never
reached them: both loops kept a private MAX_TOKENS_PER_RESPONSE = 4096 —
HALF the pre-#242 default — on the largest units in the scan. On the run
that filed #290, 31 of 182 incomplete verifications self-declared
'finish call truncated at max_tokens'. A truncated finish is deliberately
downgraded to verification-incomplete (FN-safe), so the undersized cap
silently converted adjudications into needs_review.

Fix: extract llm.helpers.DEFAULT_MAX_TOKENS (20000, the #242 rationale
comment moved with it); simple_text's signature default and both
agent-loop constants now reference the SAME constant, so the two budget
paths cannot drift apart again. Zero behavior change for simple_text
callers; verify/enhance per-turn cap rises 4096→20000 (a CAP, not a floor
— models still stop at end_turn).

Scope notes (deliberate deviations from the issue's suggestions, with
reasons):
- suggestion 1's 'read from configuration' and suggestion 3's PhaseBinding
  property are architecture/config-surface changes, not fixes — deferred;
  the shared constant delivers the anti-drift property they were after.
- suggestion 4's full 16-site census guard is NOT added: the other 4096
  sites (llm_reachability, report generator, context_enhancer, etc.) are
  single-shot simple_text-style calls with their own cost/shape tradeoffs
  this issue did not analyze; blanket-justifying them would be guessing.
  The drift test covers the agent loops the issue indicts.
- #291 (the input cap on appended tool results) is the sibling gap,
  independent, and remains open.

Counts (commands run at commit time):
- pytest tests/test_issue290_output_budget.py tests/test_llm_helpers.py tests/test_llm_helpers_unit.py -q → 24 passed
- mutation smoke: fix reverted → test file errors (import DEFAULT_MAX_TOKENS fails) — RED proof; re-applied → green
- hermeticity oracle: env -i HOME=/tmp/fakehome-noconfig → 3 passed
- pytest tests/ -q → 3152 passed, 2 failed (pre-existing zig local-env,
  stash-verified; CI green with them), 32 skipped
- ruff check . → clean

Fixes #290

from utilities.agentic_enhancer import agent as enhancer_agent # noqa: E402
from utilities import finding_verifier # noqa: E402
from utilities.llm import helpers as llm_helpers # noqa: E402
@gadievron
gadievron merged commit ac5c760 into master Aug 28, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verify calls adapter.complete with max_tokens=4096, bypassing the simple_text default raised in #242

1 participant