inference-backends.md

Inference backends

Hive can route agents through OpenAI-compatible model gateways instead of a subscription CLI model. The supported gateway backend IDs are vllm, llm-d, litellm, watsonx, and named Model Gateways such as openrouter.

See Gateway-path tier for which dashboard/agent features (terminal access, model discovery, token metering, login) are guaranteed, best-effort, or not applicable on this path, and what a gateway smoke check must prove.

Looking for Copilot (or Claude, Codex, Gemini…)? It is not a Model Gateway

Subscription CLI backends — copilot, claude, codex, gemini, goose, and friends — do not appear in Governor Config → Model Gateways, and that is expected: a gateway is an OpenAI-compatible HTTP endpoint that Hive routes requests to, while a CLI backend is a separate agent binary that brings its own subscription auth. If you came here trying to make your hive use GitHub Copilot, configure it as an agent backend instead — Copilot is an agentic coding CLI that runs the agent itself, not an inference endpoint Hive routes model calls to:

1. Set backend: copilot on the agent (per-agent, in Agents config or YAML — see agent-configuration.md).
2. Authenticate once per hive: click the Login button on any copilot-backend agent's card in the dashboard. It runs a GitHub device-flow login (POST /api/copilot-auth/start) — enter the one-time code at github.com/login/device. The token is stored durably on the hive (/data/copilot-user-token) and shared by all copilot agents. Alternatively, provide a token via the COPILOT_GITHUB_TOKEN env var (see env-vars.md).
3. Verify the login actually activated. A saved token is not proof the account can run inference: the device flow can fail server-side, and an org policy can refuse the CLI integration even when the seat is valid. The login dialog now verifies the seat before it reports success, and you can re-check at any time with the Re-verify button or POST /api/copilot-auth/verify. GET /api/copilot-auth/status carries the same verdict under seat.
4. Pick a model: Hive probes your plan's live, entitlement-filtered Copilot /models list, so the agent's model dropdown shows exactly what your Copilot subscription can use.

Verifying your Copilot login

logged_in: true means only that a token is stored. The seat.state field is the answer that matters, and each value has one cause and one fix:

seat.state	What GitHub said	What to do
active	200 — the seat is live	Nothing; agents can run inference.
no_seat	403 "not licensed to use Copilot"	The account genuinely has no seat. Check github.com/settings/copilot, or log in with an account that has one.
blocked	403 that is not a licence verdict	Usually an org policy blocking the integration ID, not a seat problem. Ask an org owner to allow the Copilot CLI integration, or set HIVE_COPILOT_INTEGRATION_ID to one your org permits.
rejected	401	The login never fully activated. Run the login again and approve the device code before it expires.
unreachable	network failure or 5xx	Transient or hive-side. It says nothing about the account — retry.

Every verdict also carries credential, naming which credential was verified — the dashboard login, or COPILOT_GITHUB_TOKEN. This matters because a hive can hold more than one: if the verdict names a credential you did not expect, that mismatch is the bug, not the seat.

Self-checks from a shell on the hive:

# Does this token have a Copilot seat at all?
gh api /copilot_internal/user

# What did the hive log at login time?
#   "Copilot CLI authenticated via device flow"                       -> activated
#   "... authenticated via device flow but seat verification failed"  -> see seat= and detail=

If model names in the dashboard carry a (Copilot seat not licensed) suffix, read the seat verdict first: that label is the catalog probe's verdict and can be about a different credential than the one you just logged in with.

Full per-backend setup details (install, auth, confinement) are in backend-setup.md.

How routing works

The agent still launches Claude Code in bare mode. Hive writes Claude settings that point ANTHROPIC_BASE_URL at Hive's local translator, then the translator converts Anthropic Messages API calls to OpenAI-compatible requests and forwards them to the selected gateway. The backend name selects the upstream route; it is not a separate agent binary.

Configure a gateway

Use the dashboard's Governor Config → Model Gateways UI, or YAML. A gateway needs a name, kind, endpoint, optional key reference, and optional default model. Agents can then set backend: to either the built-in kind (vllm, llm-d, litellm, watsonx) or to a configured gateway name such as openrouter.

Each gateway also accepts an optional key_name — a human-chosen LABEL for the configured key ("Team inference key", "andy personal", …). It is safe-to-show metadata, not a secret: it records WHICH key a gateway is set to use so operators can tell keys apart without ever seeing the value. The dashboard's gateway row displays it as "Using key: <name>", or "(unnamed)" when no label is set. key_name mirrors the bob backend's KeyName field and is optional on every gateway kind (litellm / openrouter / watsonx / vllm); omitting it keeps existing gateways byte-identical in hive.yaml.

LiteLLM also has a dedicated config block:

governor:
  litellm:
    endpoint: https://litellm.example.com
    api_key_env: HIVE_LITELLM_API_KEY
    api_key_file: /secrets/litellm_api_key
    default_model: gpt-4o
    ca_bundle: /secrets/litellm-ca.pem
    local_proxy: false

Endpoint environment overrides used by v2 include:

• HIVE_VLLM_ENDPOINT for vllm
• HIVE_LLMD_ENDPOINT for llm-d
• HIVE_LITELLM_ENDPOINT for LiteLLM
• HIVE_LITELLM_API_KEY or api_key_file for LiteLLM bearer auth
• HIVE_LITELLM_MODELS as a comma-separated fallback model list when discovery fails

Never put the key value itself in YAML.

Which endpoint and key agents on the litellm backend actually use

When a gateways: entry named litellm exists (the Model Gateways tab creates one), agents on the built-in litellm backend take endpoint, default model, key and CA bundle from that gateway, and the legacy governor.litellm block / HIVE_LITELLM_ENDPOINT are ignored for routing. Only when no such gateway has an endpoint does the legacy block (env var first, then YAML) route traffic. local_proxy: true wins over both. Every Test button prints the resulting route — Agents on backend "litellm" send inference to POST <url> … — route source: … — and warns when that is not the endpoint or key the test just exercised, so "Connection OK" is never shown for an endpoint no agent will use.

The legacy governor.litellm block (the "legacy endpoint" these messages name) is edited in the dashboard under Settings → Model Gateways → Legacy LiteLLM endpoint, below the gateway list. Saving the gateway named litellm with a new endpoint repoints the legacy endpoint, default model and key store to match, so the two cannot drift apart again; clear the legacy endpoint there if it names a gateway you no longer use.

OpenRouter scan-to-fund gateway

OpenRouter is both a Model Gateway kind (kind: openrouter) and a guided funding flow. It routes through OpenRouter's OpenAI-compatible API at https://openrouter.ai/api/v1, so agents use it like any other gateway after a key is stored.

Operator setup with Model Gateways

1. Open Governor Config → Model Gateways.
2. Add a gateway named openrouter with kind openrouter. The UI preset fills the endpoint as https://openrouter.ai/api/v1.
3. Store the key as a secret value in the UI, or point api_key_env / api_key_file at a key that already exists. Hive stores only the file path or env-var name in hive.yaml.
4. Pick a default_model from discovery or enter an OpenRouter model id manually. The curated fallback default is deepseek/deepseek-chat.
5. Assign an agent with backend: openrouter (the gateway name) and either leave model: empty to use the default or set an explicit OpenRouter model id.

Equivalent YAML looks like:

governor:
  gateways:
    - name: openrouter
      kind: openrouter
      endpoint: https://openrouter.ai/api/v1
      api_key_file: /data/secrets/gateway_openrouter_api_key
      default_model: deepseek/deepseek-chat
      key_name: openrouter-prod-key  # optional: audit label shown in the dashboard, not a secret

agents:
  guide:
    backend: openrouter
    model: deepseek/deepseek-chat

Scan-to-fund flow

The dashboard can start an OpenRouter OAuth PKCE flow from the OpenRouter funding card. A sponsor picks a default model, scans the QR code or opens the returned openrouter.ai/auth link, authorizes on OpenRouter, and returns to /openrouter/callback. Hive exchanges the code for a user-controlled OpenRouter API key and upserts the openrouter gateway.

For a spoke dashboard, the key is written to the per-gateway secret-file store on that hive's PVC and hive.yaml records only api_key_file. For a hub-funded hive, the hub queues a pending gateway named openrouter and delivers it over the existing TLS heartbeat response; heartbeat-only/firewalled spokes therefore receive funding without the hub POSTing into the cluster. The spoke confirms arrival by reporting configured gateway names, after which the hub stops offering the pending secret. The hub does not persist or display the key.

Credit display and quotas

A spoke with an OpenRouter key proxies OpenRouter's /api/v1/key credit endpoint and displays limit, usage, and remaining credit without returning the key. The hub can only show whether a funded gateway is still pending delivery; after delivery, credit is read from the spoke because the hub no longer has the key.

Model discovery

Hive probes /v1/models on OpenAI-compatible gateways. LiteLLM discovery includes bearer auth when a key is configured. If discovery fails, the UI falls back to static or configured model lists and marks entries as unverified; fallback data should not be treated as proof that the endpoint is healthy.

Guided example: Watsonx via gateway

The watsonx path uses the same gateway machinery: configure the gateway endpoint and credentials, verify /v1/models, then assign an agent to that backend/gateway. Use it as the guided flow for any enterprise gateway:

1. Create the gateway in the Model Gateways UI.
2. Store the key in a secret file or env var reference.
3. Click model discovery and choose an entitlement-visible model.
4. Assign a low-risk agent and run one manual kick.
5. Check agent logs for route/model passthrough and upstream HTTP errors.

Common failures

Symptom	Likely cause	Fix
401 or repeated gateway auth errors	Missing/stale API key or wrong LiteLLM virtual key	Rotate the key, update the env/file reference, restart or reload the hive.
Test says Connection OK but agents fail with 401 Invalid proxy server token naming the correct key hash	The route the agents use and the endpoint the test probed were not the same: a stale governor.litellm.endpoint / HIVE_LITELLM_ENDPOINT still pointed at a previous gateway while the Model Gateways tab's litellm gateway (and its key) pointed at the new one, so the new key was sent to the old gateway.	Fixed: the explicit litellm gateway now wins for the endpoint as it already did for the key. Read the Agents on backend "litellm" send inference to POST … line under the Test result; if it carries a ⚠, reconcile the two sources it names.
Model dropdown empty or unverified	/v1/models unreachable or blocked	Check endpoint URL, network policy, TLS CA bundle, and gateway logs.
Test connection says a proxy refused the request	The Hive pod reached an HTTP proxy, ingress, WAF, VPN boundary, or load balancer in front of the gateway before LiteLLM evaluated the key. The diagnostic quotes the observed Server header (for example awselb/2.0) and content type; that identifies the fronting proxy class, not its owner or rules.	Allow the Hive cluster's egress IPs/network path through the gateway's ELB/WAF/VPN/allow-list, or configure an endpoint reachable from the cluster. This is outside Hive's control; compare the dashboard's HTTPS_PROXY/NO_PROXY egress note with the path your laptop uses.
Agent starts but every prompt fails	Endpoint does not implement OpenAI chat completions for the selected model	Select a chat-capable model or fix gateway routing.
Connection refused / timeout	Service name or port is wrong, or NetworkPolicy blocks it	From the Hive pod, curl the gateway health and /v1/models endpoints.
Model rejected despite discovery	The agent model differs from the gateway entitlement name	Use the exact model ID returned by /v1/models; Hive passes it through verbatim.
Agents fail with 403 team not allowed to access model … Tried to access <some other model> while the gateway card shows an entitled default_model	The agent entry has its own model: (often left over from an earlier copilot/claude backend, e.g. claude-sonnet-4-6) and a per-agent model always beats the gateway default.	Fixed: once the gateway reports its entitled set (key-info probe or that 403), a per-agent model the team cannot use is replaced per-request by the gateway default_model and a inference model not entitled on gateway warning names the agent. To make it permanent, open the agent's Configure dialog and pick "(backend default — )" in the Model select (that is "no per-agent model"; the option names the gateway default it resolves to) or one of the listed ids — options show full gateway ids and a current model outside the entitled set is marked "(current — not allowed by gateway)". The opened card's form Test now prints the same endpoint / default model / key SHA / route line as the saved-card Test. A model the ACMM pack seeded (pack-owned, not picked by an operator) is also cleared on the pack apply that runs at every restart once the agent is on a gateway backend other than the pack's, instead of being re-seeded.

See also src/deploy/inference/README.md for the sample in-cluster vLLM-compatible deployment.