feat: Multi-Key API Key Rotation with Automatic Failover (KeyPool)

[ariel42]

Summary

This feature introduces KeyPool, a transparent multi-key rotation and failover system for all API providers. Users can supply multiple API keys per provider via comma-separated environment variables, and Claudish will automatically distribute requests across keys with intelligent failover on errors.

# Before: single key, single point of failure
export GEMINI_API_KEY="key1"

# After: multiple keys, automatic rotation on errors
export GEMINI_API_KEY="key1,key2,key3"

No CLI changes, no configuration flags — just add more keys to the same environment variable.

Motivation

API rate limits (HTTP 429) are a common friction point when using LLM providers heavily. When a key is rate-limited, the user must wait or manually switch keys. Similarly, expired or revoked keys cause hard failures with no recovery path. Multi-key rotation addresses this:

• Rate limit resilience: When one key hits 429, the next key is tried immediately
• Key lifecycle management: Expired or invalid keys are skipped automatically
• Zero-downtime key rotation: Add a new key to the list, remove old ones later
• Team and CI use cases: Distribute load across multiple API keys to stay within per-key quotas

Architecture

KeyPool Class

KeyPool is the core abstraction, located at handlers/shared/key-pool.ts:

• Initialization: Parses a comma-separated key string, trims whitespace, filters empty entries
• Round-robin rotation: Keys are tried in order; the index sticks on success and advances on failure
• executeWithFailover(fetchFn): The main entry point — wraps a fetch function with automatic key rotation:
  • On rotatable HTTP errors (401, 402, 403, 408, 429, 500–504): rotates to the next key and retries
  • On body-detected invalid keys (non-rotatable status codes like 400): inspects the response body for provider-specific invalid key patterns before giving up
  • On network errors (thrown exceptions): rotates and retries
  • On non-retryable errors (e.g., 400 validation errors): returns immediately without wasting remaining keys
  • On all keys exhausted: returns the last failed response (preserving body for error details) or re-throws the last network error
• Resource safety: Drains previous response bodies to prevent connection and memory leaks, while preserving the last response for the caller

Response Body-Based Invalid Key Detection

Some providers return non-standard status codes for invalid API keys. For example, Gemini returns HTTP 400 (not normally retryable) with reason: "API_KEY_INVALID" in the response body. The isInvalidApiKeyResponse() method clones the response and inspects the body for known patterns:

Provider	Status	Detection Pattern
Gemini	400	error.details[].reason === "API_KEY_INVALID"
OpenAI	401	error.code === "invalid_api_key"
Anthropic	401	error.type === "authentication_error"
Generic	varies	Message contains "API key not valid", "invalid api key", "invalid credentials", "incorrect api key"

This provides defense-in-depth: even if a provider changes their error status code, the body-level check catches it. Non-JSON responses gracefully return false (no false positives).

Handler Integration

All 7 provider handlers have been updated to use KeyPool:

Handler	Provider	Key Env Var
base-gemini-handler.ts	Gemini	GEMINI_API_KEY
openai-handler.ts	OpenAI	OPENAI_API_KEY
anthropic-compat-handler.ts	Anthropic, MiniMax, Kimi	Provider-specific
openrouter-handler.ts	OpenRouter	OPENROUTER_API_KEY
ollamacloud-handler.ts	OllamaCloud	OLLAMA_API_KEY
litellm-handler.ts	LiteLLM	LITELLM_API_KEY
remote-provider-handler.ts	Generic remote providers	Registry-defined

Each handler wraps its fetch() call inside keyPool.executeWithFailover(), with the API key injected per-attempt rather than once at construction time.

Backward Compatibility

• Single key: KeyPool works transparently with a single key (no rotation, no extra logging)
• No key: Handlers that don't require keys continue to work unchanged
• Logging: Multi-key rotation is only logged when more than one key is configured, keeping single-key output clean

Test Coverage (116 tests)

Unit Tests (tests/key-pool.test.ts — 1,990 lines)

• Core KeyPool: Initialization (empty, single, multi-key, whitespace handling), round-robin rotation, index advancement, reset
• executeWithFailover: Success on first try, failover through keys, all-keys-exhausted (both HTTP and network errors), non-retryable status passthrough, response body draining, mixed error types
• Status code handling: Every code in ROTATABLE_STATUS_CODES verified, non-rotatable codes confirmed as passthrough
• Invalid key detection: Every detection pattern (Gemini, OpenAI, Anthropic, generic messages), negative cases (normal 400, non-JSON body), body preservation via clone, all-keys-exhausted with body detection
• Concurrency resilience: Concurrent executeWithFailover calls, large key pools (10+ keys), key exhaustion ordering

Live Provider Canary Tests (5 tests)

Real HTTP requests to Gemini, OpenAI, Anthropic, OpenRouter, and OllamaCloud with deliberately invalid API keys. Each response is piped through isInvalidApiKeyResponse() to verify the detection logic matches what the provider actually returns. If any provider changes their error format, the corresponding canary test fails — alerting maintainers to update the detection logic. This eliminates the risk of silent regression from provider API contract changes.

Files Changed (37)

• packages/cli/src/handlers/shared/key-pool.ts — new KeyPool class
• packages/cli/src/handlers/shared/gemini-retry.ts — Gemini-specific retry integration
• packages/cli/src/handlers/shared/remote-provider-handler.ts — generic remote provider KeyPool integration
• packages/cli/src/handlers/*.ts — all 7 handlers updated to use KeyPool
• tests/key-pool.test.ts — 116 tests (1,990 lines)
• tests/gemini-retry.test.ts — Gemini retry tests
• Synced to packages/core/ and src/

[claudish-bot]

@ariel42 This is a well-thought-out proposal. Currently every handler stores a single apiKey string and passes it straight to fetch with no retry or rotation — so this would be net-new functionality across all 7+ handlers.

A few things worth discussing before merging 37 files:

• Scope of executeWithFailover — wrapping every handler's fetch call is a big surface area change. Have you considered a middleware/interceptor approach in the proxy layer (src/proxy-server.ts) instead? That'd be one integration point rather than seven.

• Interaction with existing retry logic — gemini-retry.ts in handlers/shared/ already does exponential backoff for Gemini. How does KeyPool's rotation compose with that? Does KeyPool retry before the existing retry, or replace it?

• Canary tests hitting real APIs — the 5 live provider tests are clever for detecting contract drift, but they'll fail in CI without valid (deliberately invalid) keys configured. What's the plan for CI environments?

The backward-compat story (single key = no rotation, no extra logging) is the right call. Would like to see this as a draft PR so we can review the handler integration pattern on one handler before all seven.

[ariel42]

Thanks for the review — great questions. This feature is fully implemented and ready to merge in PR #61. Let me address each point:

Middleware vs per-handler integration

A proxy-layer interceptor was considered, but key rotation needs to happen at the handler level because each provider has different authentication mechanisms: Gemini uses a query parameter (?key=), OpenAI/OpenRouter/OllamaCloud use Authorization: Bearer, and Anthropic uses x-api-key. The key injection is tightly coupled to how each handler constructs its request, so a single middleware can't inject keys uniformly. The tradeoff is 7 integration points, but each is a small, mechanical change — the fetch call is simply wrapped in keyPool.executeWithFailover(async (key) => { ... }), keeping the handler's existing logic intact.

Interaction with gemini-retry.ts

KeyPool and the Gemini retry logic operate at different layers with clear separation. gemini-retry.ts handles Gemini-specific exponential backoff for transient errors (503, quota exhaustion) on a single key. KeyPool operates above that — it rotates to a different key when the current key fails. The composition is: KeyPool tries key1 → Gemini handler (with its own retry) → if the key itself is bad (401/400 with API_KEY_INVALID), KeyPool rotates to key2 → Gemini handler retries with key2. They don't conflict because KeyPool rotates on key-specific errors, while gemini-retry retries on transient server errors.

Canary tests in CI

The live canary tests use deliberately invalid API keys (the literal string "invalid-key-000-canary-test") — no real credentials needed. They only require outbound HTTPS access to the provider endpoints. In a fully offline CI environment they'd fail, but any CI that can reach the public internet will run them correctly. If offline CI is a concern, they could be tagged and excluded from offline runs, though we'd recommend keeping them in the default suite to catch provider contract changes early.

[claudish-bot]

@ariel42 Thanks for the thorough answers — the per-handler approach makes sense given the auth divergence (?key= vs Authorization: Bearer vs x-api-key), and the layering with gemini-retry.ts is clean.

The invalid-key canary approach is clever — no secrets needed, just outbound HTTPS. That's a reasonable default for most CI setups.

I'd say this is ready for @jackrudenko to review PR #61 when he gets a chance — the design decisions are well-justified.

[erudenko]

This feature request is now tracked on our public feedback board: Vote for Multi-Key API Rotation

Architecture design is complete — lazy header evaluation in ComposedHandler enables key rotation with minimal changes (~5 files, ~50 lines). PR #61's KeyPool class is directly reusable.