Skip to content

Feature: Avoid unnecessary account switching to preserve provider prompt cache #12

Description

@thelioo

Feature: Avoid unnecessary account switching to preserve provider prompt cache

Problem

The balancer currently switches between accounts/providers more often than necessary. Each switch breaks the provider's prompt cache, because caching is tied to a stable per-account context (e.g. prompt_cache_key). When the active account changes mid-conversation, the next request can no longer reuse the cached prompt prefix, so it gets billed as fresh input tokens instead of cached ones.

For multi-turn coding sessions this is expensive, the conversation context that should have been a cache hit is recomputed from scratch every time the account rotates.

Proposed behavior

Make account selection "sticky": once a request lands on a given account within a conversation/session, keep routing subsequent requests to that same account as long as it's healthy, and only switch when there's a real reason to.

Switch only when:

  • The current account hits a rate limit / quota error
  • The current account returns an auth or hard error
  • The account is otherwise unavailable

Stay on the current account when:

  • It's still within limits and responding normally
  • The only "reason" to switch would be plain round-robin / load distribution

Metadata

Metadata

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions