Version: eve 0.39.1, Slack channel, @workflow/world-postgres backend
What happens
When the model provider returns an error mid-turn — in our case Anthropic's Your credit balance is too low to access the Anthropic API — eve emits turn.failed followed by session.failed, and the default Slack handler posts:
This session couldn't recover from an error (Model provider API error: Your credit balance is too low…).
Start a new thread to continue — I can't pick this one back up.
The session is permanently dead. After we topped up credits minutes later, the thread — an active multi-person product discussion with a pending approved tool call — could not be resumed; every affected thread across the workspace had to be restarted, losing conversational context (durable facts survived only where the agent's own memory tooling had captured them).
Why this seems wrong
Billing exhaustion, provider outages, and 429 storms are environmental and transient: nothing about the session's own state is corrupted. Terminal failure is proportionate for unreplayable/corrupted state, but here the session would work perfectly on the next attempt. The blast radius is also unbounded — a single billing lapse terminally kills every active session in the deployment at once.
Notably, the durable machinery already behaves better than the session lifecycle: an approved tool call from the same turn executed successfully (our GitHub comment posted), yet the session that produced it was destroyed.
Expected / proposed
- Classify provider API errors (auth/billing/429/5xx) as turn-fatal, session-recoverable: fail the turn with the existing error message, keep the session; the next inbound message retries on a fresh turn.
- Or make the escalation policy configurable (e.g.
session.failed only after N consecutive failed turns, or a recoverable predicate on provider errors).
Repro
- Run any Slack-connected agent with a direct Anthropic key.
- Exhaust the key's credit balance (or block egress to the provider).
- Send a message: observe
turn.failed → session.failed, session unrecoverable.
- Restore the provider; the thread cannot continue.
Error id from our incident, if useful for tracing: 3d7ee31c-8b36-481e-b1a4-15f940414141 (2026-08-20).
Version: eve 0.39.1, Slack channel,
@workflow/world-postgresbackendWhat happens
When the model provider returns an error mid-turn — in our case Anthropic's
Your credit balance is too low to access the Anthropic API— eve emitsturn.failedfollowed bysession.failed, and the default Slack handler posts:The session is permanently dead. After we topped up credits minutes later, the thread — an active multi-person product discussion with a pending approved tool call — could not be resumed; every affected thread across the workspace had to be restarted, losing conversational context (durable facts survived only where the agent's own memory tooling had captured them).
Why this seems wrong
Billing exhaustion, provider outages, and 429 storms are environmental and transient: nothing about the session's own state is corrupted. Terminal failure is proportionate for unreplayable/corrupted state, but here the session would work perfectly on the next attempt. The blast radius is also unbounded — a single billing lapse terminally kills every active session in the deployment at once.
Notably, the durable machinery already behaves better than the session lifecycle: an approved tool call from the same turn executed successfully (our GitHub comment posted), yet the session that produced it was destroyed.
Expected / proposed
session.failedonly after N consecutive failed turns, or arecoverablepredicate on provider errors).Repro
turn.failed→session.failed, session unrecoverable.Error id from our incident, if useful for tracing:
3d7ee31c-8b36-481e-b1a4-15f940414141(2026-08-20).