Skip to content

feat: optional bearer auth for local providers (mlx, llamacpp, lmstudio, ollama) - #9

Open
vinayvobbili wants to merge 1 commit into
stevibe:mainfrom
vinayvobbili:feat/local-provider-api-keys
Open

feat: optional bearer auth for local providers (mlx, llamacpp, lmstudio, ollama)#9
vinayvobbili wants to merge 1 commit into
stevibe:mainfrom
vinayvobbili:feat/local-provider-api-keys

Conversation

@vinayvobbili

Copy link
Copy Markdown

Summary

Adds optional <PROVIDER>_API_KEY env-var support to the four local provider kinds (mlx, llamacpp, lmstudio, ollama) so models served behind bearer auth are reachable from the CLI.

Today buildProviderApiKey() only reads OPENROUTER_API_KEY; any locally-served OpenAI-compatible endpoint deployed with --api-key (or behind an authenticating reverse proxy) silently goes out without an Authorization header and gets rejected.

Concrete deployments this unblocks

  • vllm-mlx with --api-key — recommended prod config for serving a single MLX model on a shared LAN
  • llama.cpp server with --api-key
  • LM Studio with the bearer-auth toggle enabled
  • Ollama fronted by an authenticating reverse proxy

Change shape

  • lib/models.tsbuildProviderApiKey now also reads MLX_API_KEY, LLAMACPP_API_KEY, LMSTUDIO_API_KEY, OLLAMA_API_KEY (in addition to OPENROUTER_API_KEY). The local-provider keys are optional — when unset, the request goes out exactly as it does today (no Authorization header), so the default unauthenticated local-serving path is unchanged.
  • .env.example — documents the new vars next to the existing host vars.

Symmetric with the desktop app's api_key / api_key_env provider fields in CONFIG_SCHEMA_V1.md — the desktop UI has supported per-provider api keys for a while; this brings the CLI in line.

Validation

Ran the full 15-scenario bench against a vllm-mlx server hosting Qwen3-Coder-30B-A3B-Instruct-8bit with --api-key set:

export MLX_HOST=http://my-host:8003
export MLX_API_KEY=...
export LLM_MODELS=mlx:qwen3-coder-30b-a3b
node dist-cli/cli/run.js --temperature 0 --timeout 180

Result: 83 / 100 — ★★★★ Good (12 pass / 1 partial / 2 fail in ~39 s wall-clock). Before this patch, every request returned Invalid API key.

Notes

  • Adjacent to Add OpenCode GO & vLLM Routing #6 (which adds a separate vllm provider kind). Different scope — that PR adds a new provider; this PR makes the existing local providers usable with bearer-protected servers. They don't conflict on lib/models.ts rebases other than the buildProviderApiKey function body.
  • No behavior change for users who don't set the new env vars.

…io, ollama)

The four local-provider kinds all support OpenAI-compatible bearer auth
in their server config, but the CLI's buildProviderApiKey() only honored
OPENROUTER_API_KEY — meaning any locally-served model deployed behind
--api-key (or a reverse-proxy with bearer auth) was unreachable from
ToolCall-15 without a code change.

Concrete deployments this unblocks:

- vllm-mlx with `--api-key` (recommended prod config for the OpenAI-
  compatible endpoint on a shared LAN)
- llama.cpp server with `--api-key`
- LM Studio with the bearer-auth toggle enabled
- Ollama fronted by an authenticating reverse proxy (we run one of these
  internally; the pattern is common enough to deserve a first-class env var)

Each provider now reads an optional `<PROVIDER>_API_KEY` env var. When
unset, the request goes out without an Authorization header — exact
behavior the CLI has today, so this is purely additive for the default
unauthenticated local-serving path.

Symmetric with the desktop app's `api_key` / `api_key_env` provider
fields documented in CONFIG_SCHEMA_V1.md.

Validation: ran the full 15-scenario bench against a vllm-mlx server
hosting Qwen3-Coder-30B-A3B-Instruct-8bit with --api-key set; MLX_API_KEY
correctly attaches the bearer header and all scenarios that were
expected to reach the model did.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant