Feature: Avoid unnecessary account switching to preserve provider prompt cache
Problem
The balancer currently switches between accounts/providers more often than necessary. Each switch breaks the provider's prompt cache, because caching is tied to a stable per-account context (e.g. prompt_cache_key). When the active account changes mid-conversation, the next request can no longer reuse the cached prompt prefix, so it gets billed as fresh input tokens instead of cached ones.
For multi-turn coding sessions this is expensive, the conversation context that should have been a cache hit is recomputed from scratch every time the account rotates.
Proposed behavior
Make account selection "sticky": once a request lands on a given account within a conversation/session, keep routing subsequent requests to that same account as long as it's healthy, and only switch when there's a real reason to.
Switch only when:
- The current account hits a rate limit / quota error
- The current account returns an auth or hard error
- The account is otherwise unavailable
Stay on the current account when:
- It's still within limits and responding normally
- The only "reason" to switch would be plain round-robin / load distribution
Feature: Avoid unnecessary account switching to preserve provider prompt cache
Problem
The balancer currently switches between accounts/providers more often than necessary. Each switch breaks the provider's prompt cache, because caching is tied to a stable per-account context (e.g.
prompt_cache_key). When the active account changes mid-conversation, the next request can no longer reuse the cached prompt prefix, so it gets billed as fresh input tokens instead of cached ones.For multi-turn coding sessions this is expensive, the conversation context that should have been a cache hit is recomputed from scratch every time the account rotates.
Proposed behavior
Make account selection "sticky": once a request lands on a given account within a conversation/session, keep routing subsequent requests to that same account as long as it's healthy, and only switch when there's a real reason to.
Switch only when:
Stay on the current account when: