Add fallback LLM provider on timeout/rate-limit

[mrunmayeekokitkar]

Summary

Implements a fallback LLM provider mechanism that automatically retries requests with a secondary provider when the primary provider encounters recoverable failures. This improves reliability during rate limiting, transient server errors, and request timeouts while allowing users to configure fallback providers directly from the AI settings page.

Related Issue

Closes #34

---

Changes

LLM Provider

Updated provider.ts to support automatic fallback providers.

Configuration

Extended the LLMConfig interface with:

• fallbackProvider
• fallbackApiKey
• fallbackModel

Fallback Logic

Implemented automatic fallback for both:

• generateText()
• streamText()

The primary provider is attempted first. If a recoverable error occurs, the request is automatically retried using the configured fallback provider.

Fallback Conditions

Fallback is triggered for:

• HTTP 429 (rate limiting)
• HTTP 5xx server errors
• Request timeout (AbortError)

Fallback is not triggered for:

• Other 4xx client errors (except 429), since these generally indicate user or request configuration issues.

Timeout Handling

Added fetchWithTimeout() to enforce a 30-second timeout for provider requests.

Provider Logging

Added provider logging to identify which provider handled each request.

Example log output:

[PRIMARY] Using LLM provider: OpenAI
[FALLBACK] Using LLM provider: Groq

This improves debugging and operational visibility.

---

AI Settings

Updated ai-settings.tsx to support fallback provider configuration.

Users can now:

• Enable or disable a fallback provider
• Select a fallback provider
• Configure a fallback API key
• Select a fallback model

Fallback configuration is stored alongside the primary LLM configuration.

---

Environment Configuration

Updated .env.local.example by adding:

• GROQ_API_KEY

This provides an example configuration for using Groq as a secondary LLM provider.

---

Testing

Created:

• llm-fallback.test.ts

Added 8 comprehensive tests covering:

• Successful primary provider requests
• 429 rate limit fallback
• 5xx server error fallback
• Timeout fallback (AbortError)
• Non-recoverable 4xx errors
• Logging behavior
• Primary/fallback provider selection
• Overall fallback workflow

All tests pass successfully.

---

Fallback Behavior

Fallback occurs for:

• 429 Rate Limit
• 5xx Server Errors
• Request Timeout (AbortError)

Fallback does not occur for:

• 400 Bad Request
• 401 Unauthorized
• 403 Forbidden
• 404 Not Found
• Other non-recoverable client errors

---

Implementation Summary

provider.ts

• Extended LLMConfig with fallback configuration.
• Added shouldUseFallback() helper.
• Added logProvider() helper.
• Added fetchWithTimeout().
• Implemented fallback support in:
  • generateText()
  • streamText()

ai-settings.tsx

• Added fallback provider configuration UI.
• Added provider selector.
• Added API key input.
• Added model selector.
• Persisted fallback configuration.

.env.local.example

• Added GROQ_API_KEY.

Tests

Created:

• llm-fallback.test.ts

with comprehensive fallback coverage.

---

Checklist

• I have read the project's contributing guidelines.
• npm run lint passes successfully.
• npm run test passes successfully.
• I have tested my changes locally.
• I have added screenshots for UI changes (if applicable).
• My changes do not introduce any breaking changes.

Screenshots (if applicable)

N/A

Acceptance Criteria

• The implementation satisfies the requirements of the related issue.
• The code follows the project's coding conventions.
• Documentation has been updated where necessary.
• This pull request is ready for review.

Additional Notes

The fallback mechanism automatically improves reliability without changing the existing API surface. Requests continue to use the primary provider by default, with seamless failover only for recoverable failures such as rate limits, transient server errors, and timeouts.

[vercel]

@mrunmayeekokitkar is attempting to deploy a commit to the suyashyadav1709-9626's projects Team on Vercel.

A member of the Team first needs to authorize it.