fix(agent): an empty completion gets one repair before the turn dies - #264
Open
plombeer31 wants to merge 1 commit into
Open
fix(agent): an empty completion gets one repair before the turn dies#264plombeer31 wants to merge 1 commit into
plombeer31 wants to merge 1 commit into
Conversation
**~500 events across 200+ users, still firing on 0.4.1.**
CLI-2W 157 ev / 64 users reason=empty
CLI-2X 131 ev / 41 users reason=empty
CLI-2Z 90 ev / 35 users reason=empty
CLI-5J 73 ev / 27 users reason=empty
CLI-4R 53 ev / 25 users reason=empty
`ModelError(reason=empty)` is the largest single failure bucket in the
project. Every one of those events is a turn that ended because the model
returned nothing — once.
`detectModelFailure` flags `empty`, and `executeStepInner` throws on the
spot: before the parser sees the body, and therefore before the one-shot
repair that recovers every OTHER malformed completion — a truncated
array, a stray prelude, prose where JSON belongs, a batch that failed
validation. Only `native_tools` was exempted.
The rationale in `detect-model-failure.ts` — "re-running the same prompt
just reproduces the wall" — holds for `truncated` and `no_stop`, where
the model already spent its budget on this prefix. It does not hold here:
the repair path does not replay the prompt. It rebuilds it through
`buildToolCallRepairPrompt`, with a corrective notice and a bounded token
cap. That is a different request.
So an empty body on a grammar transport now falls through to the parser,
fails to parse, and takes the repair. Nothing new is built — the same
machinery, reached by one more case. Twice empty is still terminal: the
repair's own `detectModelFailure` throws `ModelError("empty")` exactly as
before.
`native_tools` keeps its current contract untouched. A native completion
with nothing in any channel routes through `ModelError` by design, with
its own salvage path for the reasoning-only case.
**This reverses behaviour that was pinned on purpose.**
`agent-loop.test.ts` asserted `llmCalls === 1` under the name "classifies
an empty completion as ModelError and skips parse retry". That test is
updated, not worked around: it now pins two calls and never three, and
the failure it ends on is still `category: model`. If the original
decision was load-driven rather than correctness-driven, this is the
trade to argue with — one bounded 1024-token completion against a turn
that currently just dies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Sentry shows
ModelErrorwithreason=emptyis the largest single failure bucket in the project — and it is still firing on0.4.1:CLI-2Wcategory=model,reason=emptyCLI-2Xcategory=model,reason=emptyCLI-2Zcategory=model,reason=emptyCLI-5Jcategory=model,reason=emptyCLI-4Rcategory=model,reason=emptyPlus
CLI-BandCLI-Hin the same shape — ~500+ events across 200+ users, every one of them a turn that ended because the model returned nothing once.Root cause
detectModelFailureflagsempty, andexecuteStepInnerthrowsModelErroron the spot — before the parser ever sees the body, and therefore before the one-shot repair path that recovers every other malformed completion (a truncated array, a stray prelude, prose where JSON belongs, a batch that failed validation).Only
native_toolswas exempted, viaisNativeToolsEmptyCompletionHandledByParser. On the grammar transports a single empty body ends the turn with no recovery attempt at all.The rationale in
detect-model-failure.ts— "re-running the same prompt just reproduces the wall" — holds fortruncatedandno_stop, where the model already spent its budget on this prefix. It does not hold forempty: the repair path does not replay the prompt. It rebuilds it throughbuildToolCallRepairPromptwith a corrective notice and a bounded token cap (REPAIR_MAX_TOKENS), which is a materially different request.The change
isGrammarEmptyCompletionWorthRepairinglets an empty body on a non-native_toolstransport fall through to the parser, which fails to parse it and routes into the existing one-shot repair. Nothing new is built — this is the same machinery, reached by one more case."Twice empty" stays terminal: the repair's own
detectModelFailurecheck throwsModelError("empty")exactly as before.Two things deliberately unchanged:
native_toolskeeps its current contract. A native completion with nothing in any channel routes throughModelErrorby design, pinned by "native_tools: routes 'no tool_calls and no content' through ModelError, not parse_retry". That is a documented decision, not an oversight, so this PR does not reverse it.truncated/no_stopstill fail fast, for the original reason.The gap this does not close
should-advance.tsdocumentsmodel → advance: a defective completion should fall over to the next provider. That policy is structurally unreachable today.runWithFallbackresolves as soon as the stream opens (llm-fallback-seam.ts), while the emptiness is only detectable after the stream is consumed — soadvanceFromnever sees a model defect and the chain never switches on one.Closing that means either validating completions inside the fallback attempt (impossible for the streaming seam without buffering) or giving the step executor a way to report a defect back to the chain. Both are larger than this fix and change the seam's contract, so I left them out rather than half-doing them here.
This reverses a deliberate decision — please push back if the trade is wrong
agent-loop.test.tspinned the old behaviour explicitly, under the name "classifies an empty completion as ModelError and skips parse retry" (llmCalls === 1). That test is updated, not worked around: it now pins two calls and never three, and the failure it ends on is stillcategory: model.If the original call was correctness-driven — some reason an empty grammar body cannot be repaired that I have missed — this PR is wrong and should be closed. If it was load-driven, the trade is one bounded
REPAIR_MAX_TOKENS(1024) completion against a turn that currently just dies, on the largest error bucket in production.Tests
Two new cases: an empty body followed by a good repair completes the step (2 LLM calls,
replyexecuted); an empty body followed by an empty repair still throwsModelError { reason: "empty" }. Verified non-vacuous — the first fails againstmainwith theModelErrorthrown at the pre-parser check.