Summary
_classify_llm_provider_error in apps/api/runner_sandbox/agent_driver.py misclassifies the OpenAI Agents SDK exception ModelBehaviorError("Tool <x> not found in agent <y>") as llm_model_not_configured. Users then see:
"The platform AI model is not fully configured. This is a platform-side setup gap, not something you did wrong. Retrying will not help until it is fixed. Contact support..."
when the actual problem is a worker-authoring mistake: the SKILL instructions reference a tool/app (e.g. a calendar or mail toolkit) that the worker never declared in connections, so the agent hallucinates a tool call that does not exist in its tool set.
Root cause
In _classify_llm_provider_error (around line 235), the haystack is built by concatenating the exception's module, class name, and message (around line 250):
haystack = f"{module} {class_name} {text}"_MODEL_NOT_CONFIGURED_RE (lines ~70-77) contains the alternative:
model.*not (?:found|configured|supported)
With re.IGNORECASE and .* free to span the class-name/message boundary, the pattern matches ACROSS the class name and the message. For:
agents.exceptions ModelBehaviorError Tool some_toolkit__list_events not found in agent worker
the match is literally ModelBehaviorError Tool some_toolkit__list_events not found — "Model" comes from the class name ModelBehaviorError, "not found" from the tool-not-found message. Verified:
>>> _MODEL_NOT_CONFIGURED_RE.search(
... "agents.exceptions ModelBehaviorError Tool some_toolkit__list_events not found in agent worker"
... ).group(0)
'ModelBehaviorError Tool some_toolkit__list_events not found'
Because _MODEL_NOT_CONFIGURED_RE is checked second in the cascade (line ~253), every ModelBehaviorError whose message contains "not found" (any hallucinated tool name) lands on llm_model_not_configured.
Why this is wrong
ModelBehaviorError is the Agents SDK's "the model did something unexpected" class (malformed JSON, calling a tool that doesn't exist). It is model behavior, never platform configuration. No ModelBehaviorError should ever classify as llm_model_not_configured.- The resulting message is maximally misleading: it tells the user "not something you did wrong", "retrying will not help", "contact support" — for a bug that IS in their worker definition and IS fixable in one line (declare the connection).
Impact
- Any agent-mode worker with
connections: [] whose SKILL instructions reference a connected-app toolkit (calendar, email, search, etc.) fails with the platform-not-configured message on every run.
- External evaluators / new users hitting this conclude the platform itself is broken ("their AI model isn't even configured"), when the fix is on their side.
- The message actively masks the real remediation (attach the connection / fix the tool name), generating support load instead of self-service fixes.
- Observed on multiple real runs where the model invented tool names like
<provider>__list_events, google:search, and view_file that were not in the agent's tool set. (Identifiers scrubbed; can share internally.)
Proposed fix
-
Classify tool-not-found BEFORE the regex cascade. At the top of _classify_llm_provider_error, detect ModelBehaviorError (by class name / agents.exceptions module) and/or message pattern Tool .* not found in agent and return a new code, e.g. tool_not_found. Add a matching branch in _llm_error_message with an actionable message that names the tool, e.g.:
"The agent tried to call tool <name>, which is not attached to this worker. If the tool belongs to a connected app, declare that app in the worker's connections; otherwise adjust the SKILL instructions so they only reference available tools. This is a worker configuration issue, not a platform outage."
More generally, ModelBehaviorError that isn't tool-not-found should classify as model behavior (or at most llm_provider_error), never as llm_model_not_configured.
-
Harden the haystack. Prevent cross-boundary matches for this alternative: either exclude the exception class name from the text that _MODEL_NOT_CONFIGURED_RE runs against, or use a separator the regex can't span (match module/class and message separately). Wordier but safer: \bmodel\b(?!behaviorerror) style guards are brittle; separate-field matching is the robust fix.
-
Secondary (authoring-time) suggestion: warn at worker create/validate time when SKILL instructions reference an app that exists as a workspace connection but is not declared in the worker's connections. That catches the whole class of error before any run happens.
Repro sketch
- Create an agent-mode worker with
connections: [] and a SKILL.md instructing e.g. "list today's calendar events using the calendar toolkit and print them".
- Run it. The model emits a tool call like
<calendar_toolkit>__list_events; the Agents SDK raises ModelBehaviorError: Tool <calendar_toolkit>__list_events not found in agent <worker>.
- Run fails with code
llm_model_not_configured and the "platform AI model is not fully configured" message.
Summary
_classify_llm_provider_errorinapps/api/runner_sandbox/agent_driver.pymisclassifies the OpenAI Agents SDK exceptionModelBehaviorError("Tool <x> not found in agent <y>")asllm_model_not_configured. Users then see:when the actual problem is a worker-authoring mistake: the SKILL instructions reference a tool/app (e.g. a calendar or mail toolkit) that the worker never declared in
connections, so the agent hallucinates a tool call that does not exist in its tool set.Root cause
In
_classify_llm_provider_error(around line 235), the haystack is built by concatenating the exception's module, class name, and message (around line 250):_MODEL_NOT_CONFIGURED_RE(lines ~70-77) contains the alternative:With
re.IGNORECASEand.*free to span the class-name/message boundary, the pattern matches ACROSS the class name and the message. For:the match is literally
ModelBehaviorError Tool some_toolkit__list_events not found— "Model" comes from the class nameModelBehaviorError, "not found" from the tool-not-found message. Verified:Because
_MODEL_NOT_CONFIGURED_REis checked second in the cascade (line ~253), everyModelBehaviorErrorwhose message contains "not found" (any hallucinated tool name) lands onllm_model_not_configured.Why this is wrong
ModelBehaviorErroris the Agents SDK's "the model did something unexpected" class (malformed JSON, calling a tool that doesn't exist). It is model behavior, never platform configuration. NoModelBehaviorErrorshould ever classify asllm_model_not_configured.Impact
connections: []whose SKILL instructions reference a connected-app toolkit (calendar, email, search, etc.) fails with the platform-not-configured message on every run.<provider>__list_events,google:search, andview_filethat were not in the agent's tool set. (Identifiers scrubbed; can share internally.)Proposed fix
Classify tool-not-found BEFORE the regex cascade. At the top of
_classify_llm_provider_error, detectModelBehaviorError(by class name /agents.exceptionsmodule) and/or message patternTool .* not found in agentand return a new code, e.g.tool_not_found. Add a matching branch in_llm_error_messagewith an actionable message that names the tool, e.g.:More generally,
ModelBehaviorErrorthat isn't tool-not-found should classify as model behavior (or at mostllm_provider_error), never asllm_model_not_configured.Harden the haystack. Prevent cross-boundary matches for this alternative: either exclude the exception class name from the text that
_MODEL_NOT_CONFIGURED_REruns against, or use a separator the regex can't span (match module/class and message separately). Wordier but safer:\bmodel\b(?!behaviorerror)style guards are brittle; separate-field matching is the robust fix.Secondary (authoring-time) suggestion: warn at worker create/validate time when SKILL instructions reference an app that exists as a workspace connection but is not declared in the worker's
connections. That catches the whole class of error before any run happens.Repro sketch
connections: []and a SKILL.md instructing e.g. "list today's calendar events using the calendar toolkit and print them".<calendar_toolkit>__list_events; the Agents SDK raisesModelBehaviorError: Tool <calendar_toolkit>__list_events not found in agent <worker>.llm_model_not_configuredand the "platform AI model is not fully configured" message.