Add multi-provider support via LiteLLM, removing hard dependency on O…#165
Open
getmykhan wants to merge 1 commit intokarpathy:masterfrom
Open
Add multi-provider support via LiteLLM, removing hard dependency on O…#165getmykhan wants to merge 1 commit intokarpathy:masterfrom
getmykhan wants to merge 1 commit intokarpathy:masterfrom
Conversation
…penRouter Previously, all LLM calls were routed exclusively through OpenRouter using raw httpx requests. This change introduces LiteLLM as a unified LLM client, allowing users to call provider APIs directly (OpenAI, Anthropic, Google Gemini, xAI, DeepSeek, etc.) by setting the corresponding API key in .env. OpenRouter is preserved as an automatic fallback for any model that lacks a direct provider key. Why this matters: - Lower latency: direct API calls skip the OpenRouter proxy - Lower cost: no OpenRouter markup on per-token pricing - Flexibility: users can mix and match providers - No breaking change: setting only OPENROUTER_API_KEY still works exactly as before Changes: - backend/llm_client.py: New unified LLM client using litellm.acompletion() - backend/config.py: Multi-provider routing with resolve_model() - backend/council.py: Updated imports, configurable UTILITY_MODEL - backend/main.py: FastAPI lifespan migration, startup routing table - .env.example: Documents all supported provider API keys - pyproject.toml: Added litellm>=1.60.0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add multi-provider support via LiteLLM, removing hard dependency on OpenRouter
Previously, all LLM calls were routed exclusively through OpenRouter using
raw httpx requests. This change introduces LiteLLM as a unified LLM client,
allowing users to call provider APIs directly (OpenAI, Anthropic, Google
Gemini, xAI, DeepSeek, etc.) by setting the corresponding API key in .env.
OpenRouter is preserved as an automatic fallback for any model that lacks a
direct provider key.
Why this matters:
route xAI through OpenRouter
as before
How routing works:
provider (e.g. OPENAI_API_KEY for openai/* models)
prefix is translated (e.g. gemini/ -> google/, xai/ -> x-ai/) to
match OpenRouter's naming convention
ValueError is raised telling the user which env var to set
Changes:
replacing the httpx-based openrouter.py. Same query_model() and
query_models_parallel() interface for zero-touch migration.
resolve_model(), and print_routing_info(). Model identifiers now use
litellm naming convention (e.g. "gemini/" instead of "google/"). Added
UTILITY_MODEL config for lightweight tasks (title generation).
generation now uses configurable UTILITY_MODEL instead of hardcoded
OpenRouter model name.
FastAPI lifespan context manager. Prints provider routing table on startup.
usage instructions.