Model fallback & graceful degradation for token/rate limit exhaustion

[tob-as]

Problem

Overstory currently has no fallback or retry mechanism when an agent's underlying LLM provider hits token limits, rate limits (HTTP 429), or quota exhaustion. The system operates as a process-level orchestrator and delegates all API-level concerns to the CLI runtime (e.g., Claude Code). This creates several failure modes:

Scenario	Current Behavior	Impact
API rate limit (429)	CLI retries internally; if stalled >5min, watchdog nudges → kills	Agent work lost, no respawn
Context window exhaustion	CLI does internal compaction; Overstory unaware	Agent may lose context without checkpointing
Quota/billing limit hit	Agent process dies	Watchdog marks zombie → no respawn, no model switch
Provider outage	Agent stalls indefinitely	Killed by watchdog after zombieThresholdMs

When running at scale (10-25 concurrent agents), a single provider rate limit can cascade — multiple agents hit the ceiling simultaneously, stall, and get killed by the watchdog with no recovery.

Proposed Solution

1. Model Fallback Chain in config.yaml

Allow defining ordered fallback models per role:

models:
  coordinator:
    primary: opus
    fallback: [sonnet, haiku]
  lead:
    primary: opus
    fallback: [sonnet]
  scout:
    primary: haiku
    fallback: [gemini-2.5-flash]  # cross-runtime fallback
  builder:
    primary: sonnet
    fallback: [minimax-m2.5]

When resolveModel() in src/agents/manifest.ts is called, it would return the primary model. On detected failure, the system retries with the next model in the chain.

2. Failure Detection Layer

Add API-level awareness between the runtime adapter and the watchdog:

• Runtime adapters (src/runtimes/*.ts) could expose a detectFailure() method (similar to existing detectReady()) that parses tmux pane output for known error patterns:
  • rate_limit, 429, overloaded, quota exceeded, context_length_exceeded
  • Provider-specific error signatures
• Watchdog integration: Before escalating to kill, check if the failure is recoverable via model fallback

3. Automatic Agent Respawn with Degraded Model

When the watchdog detects a token/rate limit failure:

1. Save agent checkpoint (if not already saved)
2. Kill the current session
3. Respawn with the next fallback model from the chain
4. Restore from checkpoint
5. Log the degradation event for observability

4. Per-Agent Token Budget (Optional)

Allow setting soft/hard token limits per agent role:

budgets:
  scout:
    softLimit: 50000    # warn at 50k tokens
    hardLimit: 100000   # force checkpoint + stop at 100k
  builder:
    softLimit: 200000
    hardLimit: 500000

This would require real-time token counting from transcript parsing (building on existing src/metrics/transcript.ts).

Alternatives Considered

• Do nothing: Rely on CLI-level retry. Works for transient 429s but fails for quota exhaustion and provides no cross-provider fallback.
• External proxy (LiteLLM/OpenRouter): Route all API calls through a proxy that handles fallback. Adds infrastructure complexity but works without Overstory changes. However, this doesn't solve the checkpoint/respawn problem.
• Prompt-only checkpointing: Current approach — agents are instructed to checkpoint. Unreliable since it depends on the LLM following instructions.

Context

Analysis was done by reading Overstory CLI source code (src/watchdog/, src/runtimes/, src/agents/, src/config.ts). Confirmed that:

• resolveModel() is a static one-time lookup at spawn
• Watchdog operates on OS signals only (tmux alive/dead, PID alive/dead)
• No fallback, retry, budget, or degradation keys exist in the config schema
• Checkpoint system is prompt-level only, not code-enforced

This becomes especially important for multi-provider setups (e.g., Anthropic for coordinators, Fireworks for scouts, OpenAI for builders) where different providers have different rate limits and failure modes.

[jayminwest]

Really thorough analysis, @tob-as — you clearly dug into the source. The identification of resolveModel() as a static one-time lookup and the watchdog operating on process-level observables are both exactly right.

This is a real operational gap for scale deployments, so I want to make sure we scope it in a way that lands well. A few thoughts:

Start with the fallback chain config. Extending resolveModel() to return an ordered fallback list (e.g., opus -> sonnet -> haiku) is low-risk and immediately useful. It's declarative — operators state their intent without requiring runtime-level detection. This could land quickly.

Same-runtime fallback first. Cross-runtime fallback (Claude -> Gemini) is an order of magnitude more complex — different instruction files, hook deployments, env vars, worktree artifacts to clean up. Restricting fallback chains to models within the same runtime sidesteps all of that and still covers the primary use case (cascading through Anthropic tiers).

Leverage triage instead of tmux scraping. Rather than adding detectFailure() to each runtime adapter and parsing tmux pane output for error strings, the Tier 1 triage system (src/watchdog/triage.ts) already reads session logs and classifies failures. Adding a "degrade" classification alongside the existing "retry"/"terminate"/"extend" would integrate naturally with the progressive escalation model — and it's not fragile to runtime output format changes.

Token budgets as a follow-up. Real-time token counting from transcript parsing is a distinct concern from fallback/respawn. Would be great as its own issue to keep scope manageable here.

Would you be open to narrowing this issue to proposal 1 (fallback chain config) + a "degrade" triage classification? That gives us a solid foundation to layer the detection and respawn mechanics on top of in follow-ups.

[lucabarak]

Per-capability routing (config.runtime.capabilities) gives users manual control over which runtime each agent role uses, but automatic fallback on rate limits would need error detection in each adapter plus retry/requeue logic in the orchestrator.

Feels architectural enough to warrant @jayminwest's input on the design before anyone starts building.

[Pursche]

This feature would be very useful.

On a slightly related note, how does people manage their Claude subscription usage limit (the 5 hour window) when using overstory? I just ran it with the default 25 agent limit and it hit the limit of my Pro x5 plan in like 40 minutes and locked up.

Even with a 10 agent limit I'm not getting 1 hour and 15 minutes worth of full usage out of this, which is what I would need for a x20 plan to not lock up on large projects (think web browser game).

I tried a 5 agent limit which should do it, but it spawned 4 leads and they all deadlocked because none of them could spawn an agent to actually do stuff. Do you have a workaround for that? Or do you just let it lock up (hopefully with a few tasks done) and then reset it after the rate window is fixed?

Do you perhaps get it to break it up into smaller tasks and tell it to complete X tasks and then stop (making resuming easier)?

Is there a Discord or somewhere for people to discuss and share tips on using this?

[lucabarak]

@Pursche Good question — managing usage limits with 25 concurrent agents is a real pain point. A few config knobs that help today:

Reduce concurrency:

• maxConcurrent — caps total agents running at once (default 25 is aggressive for a single subscription). Try 3–5 to start.
• maxAgentsPerLead — limits how many builders each lead can spawn simultaneously.
• maxSessionsPerRun — caps total sessions per swarm run.

# .overstory/config.yaml
agents:
  maxConcurrent: 5
  maxAgentsPerLead: 2
  maxSessionsPerRun: 10

Model tiering:
Per-capability routing lets you assign cheaper/smaller models to roles that don't need Opus:

runtime:
  default: claude
  capabilities:
    scout: gemini      # lightweight exploration
    builder: claude    # heavy lifting
    reviewer: claude   # needs accuracy

This spreads load across providers and keeps your Claude budget for the work that matters.

Automated detection (in progress):
PR #98 by @liker0704 adds rate limit detection + runtime swap — when an agent gets 429'd, the watchdog swaps it to an alternate runtime and swaps back when the limit lifts. Still under review but directly addresses this.

Community tips:
The Skool community has been discussing usage patterns — worth checking for other peoples' configs and experience with different concurrency settings.

[nicofains1]

The cascade problem you describe (rate limit hits one agent, watchdog kills it, other agents stall and get killed too) is the core challenge we built AgentWatch to solve.

Your proposed "Failure Detection Layer" in section 2 aligns with what we've implemented. The gap between watchdog PID-level liveness and API-level failure awareness is real, especially at 10-25 concurrent agents.

What AgentWatch adds:

• Heartbeat-based liveness that goes beyond PID alive/dead. Agents report health state at configurable intervals, so you detect degradation before the watchdog threshold fires.
• Cross-agent correlation: when agent A stalls from a 429, correlate() traces which downstream agents (B, C, D) are affected and identifies the root cause agent.
• Cascade replay: stored payloads at each hop let you reconstruct what happened post-mortem.

import { AgentWatch } from '@nicofains1/agentwatch';
const aw = new AgentWatch();
aw.heartbeat({ agentId: 'scout-1', status: 'healthy', metadata: { model: 'haiku' } });
// Later: aw.correlate('scout-1') -> shows cascade chain

TypeScript, self-hosted SQLite, MIT licensed. Could sit between your runtime adapters and the watchdog as the observability layer mentioned in your proposed architecture.

We wrote about the specific gaps between per-agent tracing and fleet monitoring: https://nicolasfainstein.hashnode.dev/opentelemetry-now-has-agent-span-conventions-heres-what-they-miss

[nicofains1]

@Pursche Your 5-agent deadlock is a textbook resource starvation cascade. 4 leads each waiting on a builder slot that never opens, nobody makes progress.

The config knobs @lucabarak mentioned help prevent this (lower maxConcurrent, model tiering). But detection matters too - when agents are alive but blocked, PID-level monitoring says "healthy" while nothing moves.

With heartbeat-based monitoring, you can detect the pattern: 4 agents sending heartbeats but reporting zero task completions for N minutes. That triggers an alert before you notice the lockup manually. correlate() then shows the dependency chain - which agents are waiting on which.

We hit the same problem running our own multi-agent system (7 agents, one claude -p lock). The fix is both: prevention (lower concurrency) + detection (heartbeat with completion tracking).