fix(verify): agent-loop output budgets share the simple_text thinking-era default (4096 → 20000) - #382
Merged
Merged
Conversation
…-era default finding_verifier and agentic_enhancer/agent call binding.adapter.complete directly (they need raw content blocks for the multi-turn tool conversation), so PR #242's raise of simple_text's default 8192→20000 never reached them: both loops kept a private MAX_TOKENS_PER_RESPONSE = 4096 — HALF the pre-#242 default — on the largest units in the scan. On the run that filed #290, 31 of 182 incomplete verifications self-declared 'finish call truncated at max_tokens'. A truncated finish is deliberately downgraded to verification-incomplete (FN-safe), so the undersized cap silently converted adjudications into needs_review. Fix: extract llm.helpers.DEFAULT_MAX_TOKENS (20000, the #242 rationale comment moved with it); simple_text's signature default and both agent-loop constants now reference the SAME constant, so the two budget paths cannot drift apart again. Zero behavior change for simple_text callers; verify/enhance per-turn cap rises 4096→20000 (a CAP, not a floor — models still stop at end_turn). Scope notes (deliberate deviations from the issue's suggestions, with reasons): - suggestion 1's 'read from configuration' and suggestion 3's PhaseBinding property are architecture/config-surface changes, not fixes — deferred; the shared constant delivers the anti-drift property they were after. - suggestion 4's full 16-site census guard is NOT added: the other 4096 sites (llm_reachability, report generator, context_enhancer, etc.) are single-shot simple_text-style calls with their own cost/shape tradeoffs this issue did not analyze; blanket-justifying them would be guessing. The drift test covers the agent loops the issue indicts. - #291 (the input cap on appended tool results) is the sibling gap, independent, and remains open. Counts (commands run at commit time): - pytest tests/test_issue290_output_budget.py tests/test_llm_helpers.py tests/test_llm_helpers_unit.py -q → 24 passed - mutation smoke: fix reverted → test file errors (import DEFAULT_MAX_TOKENS fails) — RED proof; re-applied → green - hermeticity oracle: env -i HOME=/tmp/fakehome-noconfig → 3 passed - pytest tests/ -q → 3152 passed, 2 failed (pre-existing zig local-env, stash-verified; CI green with them), 32 skipped - ruff check . → clean Fixes #290
gadievron
requested review from
dgeyshis,
shahar-davidson and
sounil
as code owners
August 28, 2026 20:36
|
|
||
| from utilities.agentic_enhancer import agent as enhancer_agent # noqa: E402 | ||
| from utilities import finding_verifier # noqa: E402 | ||
| from utilities.llm import helpers as llm_helpers # noqa: E402 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
finding_verifierandagentic_enhancer/agentcallbinding.adapter.complete(...)directly (they need raw content blocks for the multi-turn tool conversation), so PR #242's raise ofsimple_text's default 8192→20000 never reached them: both loops kept a privateMAX_TOKENS_PER_RESPONSE = 4096— half the pre-#242 default — on the largest units in the scan. On the run that filed #290, 31 of 182 incomplete verifications self-declared "finish call truncated at max_tokens". A truncated finish is deliberately downgraded to verification-incomplete (FN-safe), so the undersized cap silently converted adjudications into needs_review.Fix: extract
llm.helpers.DEFAULT_MAX_TOKENS(20000, the #242 rationale comment moved with it);simple_text's signature default and both agent-loop constants now reference the same constant, so the two budget paths cannot drift apart again. Zero behavior change forsimple_textcallers; the verify/enhance per-turn cap rises 4096→20000 (a CAP, not a floor — models still stop at end_turn).Note: this issue was sequenced behind #285/#286/#287 so a budget change is measurable (status honesty, error-vocabulary, fingerprint keying) — those landed in #376/#377/#378.
Scope notes (deliberate deviations from the issue's suggestions, with reasons):
PhaseBindingproperty are architecture/config-surface changes, not fixes — deferred; the shared constant delivers the anti-drift property they were after.llm_reachability, report generator,context_enhancer, etc.) are single-shotsimple_text-style calls with their own cost/shape tradeoffs this issue did not analyze; blanket-justifying them would be guessing. The drift test covers the agent loops the issue indicts.verifyappends raw tool results with no input cap, whileenhancecaps both and documents why #291 (the input cap on appended tool results) is the sibling gap, independent, and remains open.The private-run figures (31/182) quoted from the issue are provenance-labeled there; the code defect and the fix contract are fully checkable and test-covered.
Test plan
3 hermetic tests: both agent-loop constants ARE the shared constant (identity, not coincidental equality);
simple_text's signature default is that constant; no private numeric budget pins remain in the agent-loop sources.Verification evidence (commands + results)
pytest tests/test_issue290_output_budget.py tests/test_llm_helpers.py tests/test_llm_helpers_unit.py -qDEFAULT_MAX_TOKENSfails) — RED proof; re-applied → greenenv -i HOME=/tmp/fakehome-noconfig python -m pytest tests/test_issue290_output_budget.pypytest tests/ -qruff check .(CI scope)Fixes #290