selfhost(ai): LiteLLM proxy profile — multi-model routing, Redis response caching, cost dashboard

[JSONbored]

Part of #1029.

Context

Every AI provider call today goes directly from the review engine to the provider (Anthropic, OpenAI, Ollama, etc.). There is no shared routing layer, no fallback chain, no response cache, and no cost visibility. Adding a LiteLLM proxy as an optional --profile ai-proxy service changes all of that with zero code changes to the review engine — the app just points AI_BASE_URL at http://litellm:4000/v1.

What LiteLLM gives us

• Single OpenAI-compatible endpoint that routes to any backend: Ollama (local), Anthropic (Claude), OpenAI (GPT-4o), Azure, Cohere, Mistral, etc.
• Fallback chains: AI_MODEL=claude-sonnet,ollama/llama3.1 — if Anthropic is down or quota-exhausted, falls through to local Ollama automatically
• Redis semantic response cache: identical/near-identical review prompts return the cached response instantly (no tokens burned, no latency). Needs the Redis profile active.
• Per-model rate limits and budgets: cap spend per model or per day
• Cost dashboard at http://localhost:4000/ui — tracks every token used across all providers
• OpenTelemetry export — feeds into the Langfuse tracing profile (#LANGFUSE_ISSUE)

Implementation plan

docker-compose.yml additions

# --profile ai-proxy
litellm:
  image: ghcr.io/berriai/litellm:main-latest
  profiles: ["ai-proxy"]
  ports:
    - "4000:4000"
  volumes:
    - ./litellm/config.yml:/app/config.yaml:ro
  environment:
    LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
    REDIS_URL: ${REDIS_URL:-}         # enables response cache when redis profile is also active
    DATABASE_URL: ${LITELLM_DB_URL:-} # persists spend tracking (optional; uses in-memory if absent)
  command: ["--config", "/app/config.yaml", "--port", "4000"]
  depends_on:
    gittensory:
      condition: service_healthy

litellm/config.yml.example

model_list:
  - model_name: default
    litellm_params:
      model: ollama/llama3.2
      api_base: http://ollama:11434
  - model_name: claude-sonnet
    litellm_params:
      model: claude-sonnet-4-6
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  routing_strategy: least-busy
  fallbacks:
    - {"claude-sonnet": ["gpt-4o", "default"]}

litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: redis
    port: 6379

.env.example additions

# LiteLLM proxy (--profile ai-proxy)
# LITELLM_MASTER_KEY=sk-local-changeme  # protects the proxy UI
# AI_BASE_URL=http://litellm:4000/v1    # point gittensory at the proxy
# AI_MODEL=claude-sonnet                # must match a name in litellm/config.yml

server.ts / docker-compose.yml wiring

When AI_BASE_URL=http://litellm:4000/v1, the existing openai-compatible provider already works — no app-level changes needed. The proxy is transparent.

Redis caching extension (same PR)

Extend the current Redis integration beyond rate limiting:

1. GitHub API response cache (src/selfhost/redis-cache.ts): cache GET /repos/:owner/:repo, GET /pulls/:pr, GET /repos/:owner/:repo/contents/:path responses with a 60s TTL. Reduces GitHub API quota burn during heavy review periods.
2. Review result dedup: cache the gate verdict for (repo, pr_number, head_sha) with a 5-minute TTL. Prevents double-processing when a webhook fires twice for the same push.

Both are opt-in when REDIS_URL is set — fall through to live calls when Redis is absent.

Acceptance criteria

• docker compose --profile ai-proxy up starts LiteLLM proxy at port 4000
• http://localhost:4000/ui shows cost dashboard (requires LITELLM_MASTER_KEY)
• AI_BASE_URL=http://litellm:4000/v1 routes reviews through LiteLLM
• Redis response cache activated when --profile redis is also active
• Fallback chain tested: primary model unreachable → falls through to next
• GitHub API response cache implemented and covered (both arms of cache hit/miss)
• Review dedup cache implemented and covered
• litellm/config.yml.example checked in
• .env.example updated with LiteLLM vars
• npm run test:ci green, Codecov ≥ 97% patch