Skip to content

feat!: load LiteLLM models from a file, and fix version lockstep - #14

Merged
neuromaxer merged 1 commit into
mainfrom
feat/litellm-models-path
Aug 7, 2026
Merged

neuromaxer merged 1 commit into
mainfrom
feat/litellm-models-path

Conversation

@neuromaxer

Copy link
Copy Markdown
Contributor

What changed

Breaking: the LiteLLM model list now comes from one place — a JSON file named by LITELLM_MODELS_PATH.

Configuring models previously meant choosing between three overlapping tiers (LITELLM_MODELS_JSONLITELLM_MODELS csv → LITELLM_DEFAULT_MODEL alone), with four more env vars supplying the per-model values the csv tier could not express. Only the structured tier was used in practice, and it required cramming a multi-kilobyte JSON document onto a single env line — undiffable, uncommentable, and awkward for a control plane to generate from its own model catalogue.

Config surface after this change:

Env Purpose
LITELLM_BASE_URL Proxy base URL. Unset ⇒ no provider.
LITELLM_API_KEY Virtual key.
LITELLM_MODELS_PATH New. Path to a JSON array of model entries.
LITELLM_DEFAULT_MODEL Default model id; must appear in the file.
LITELLM_API Provider-wide API fallback.

Removed, and rejected at startup rather than silently ignored: LITELLM_MODELS_JSON, LITELLM_MODELS, LITELLM_CONTEXT_WINDOW, LITELLM_MAX_TOKENS, LITELLM_REASONING, LITELLM_COMPAT_JSON, LITELLM_DEFAULT_THINKING. Each error names its replacement.

Rejecting is the point. A stale value would leave the provider unregistered, which presents as "the agent has no models" with nothing indicating the cause — that is exactly how this surfaced in a real deployment. The other failure modes are loud for the same reason: an unreadable path, malformed JSON, a non-array, or an entry without an id all throw.

parseModels goes from three tiers to one. parseCompat, modelFromId, parseBool, parsePositiveInt and parseJsonObject are gone.

Presets are deliberately kept

modelPreset() is unchanged. It carries thinkingLevelMap, defaultThinkingLevel and compat.thinkingFormat for openai/gpt-5.5 and the DeepSeek models — metadata a consumer's own catalogue generally does not have. DeepSeek's null entries mark levels the model cannot do and drive clamping, so dropping them would silently degrade the thinking selector for exactly the models we serve.

File entries override presets field by field, so a bare {"id": "openai/gpt-5.5"} still gets the full preset. Merge depth drops 3 → 2 now that provider-wide compat is a fixed baseline.

Also: version lockstep was incidental, not enforced

Found while preparing this release. updateInternalDependencies propagates only downward, to packages that depend on the changed one:

agent-server ──▶ agent-protocol ◀── agent-client

agent-server is a leaf, so an agent-server-only changeset bumps only agent-server — skewing the image tag away from the npm versions. Confirmed with a changeset version dry run (agent-server 0.2.0, others 0.1.7), and by 0.1.7 itself: the bake-seccomp-profile changeset named only agent-server, and agent-client's changelog shows it bumped solely because agent-protocol did. Every release so far looked locked together by coincidence.

This PR declares the three packages as a changesets fixed group, so all three always share a version and the image tag matches. Consumers pin one version for the whole stack, so that skew would break the cheapest available check ("does the npm version match the image tag?"). Cost is a no-op bump for packages whose code did not change. Rationale is recorded in the root README.

Verified with fixed in place: all three → 0.2.0, internal deps repinned, changelogs written.

Testing

npm test155 passing. 22 new tests in test/litellmModels.test.ts cover file loading, default-model selection, preset precedence and override, every failure mode, and each removed variable. The two existing LiteLLM tests in server.test.ts are migrated to the file mechanism (one relied on the deleted "default model only" tier).

npm run check:ci, npm run typecheck, and the gen:contract freshness gate all clean.

Consumer impact

openorange is the only consumer — verified: zero LITELLM_* references in appx's Go control plane. It will move to LITELLM_MODELS_PATH with a file rendered from its own model catalogue and mounted read-only, matching how it already passes MODEL_CATALOG_PATH to operator-api. That change is prepared and lands once this releases.

BREAKING CHANGE: the LiteLLM model list now comes only from the JSON file named
by LITELLM_MODELS_PATH. Seven env vars are removed and rejected at startup.

Configuring models meant choosing between three overlapping tiers
(LITELLM_MODELS_JSON, a LITELLM_MODELS csv, or LITELLM_DEFAULT_MODEL alone),
with four more vars supplying the per-model values the csv tier could not
express. Only the structured tier was ever used in practice, and it required
cramming a multi-kilobyte JSON document onto one env line: undiffable,
uncommentable, and awkward for a control plane to generate. The list is
structured per-model config, so it belongs in a file.

Removed, and rejected with a message naming the replacement rather than ignored:
LITELLM_MODELS_JSON, LITELLM_MODELS, LITELLM_CONTEXT_WINDOW, LITELLM_MAX_TOKENS,
LITELLM_REASONING, LITELLM_COMPAT_JSON, LITELLM_DEFAULT_THINKING. Rejecting is
the point — a stale value would leave the provider unregistered, which presents
as "the agent has no models" with nothing indicating why. An unreadable path,
malformed JSON, a non-array and an entry without an id now throw for the same
reason. parseModels drops from three tiers to one; parseCompat, modelFromId,
parseBool, parsePositiveInt and parseJsonObject are gone.

modelPreset() is unchanged and still supplies the dialect for openai/gpt-5.5 and
the DeepSeek models. It is deliberately kept: it carries thinkingLevelMap,
defaultThinkingLevel and compat.thinkingFormat, which a consumer's own model
catalogue generally does not, and DeepSeek's null entries mark levels the model
cannot do and drive clamping. File entries override presets field by field, so a
bare {"id": "openai/gpt-5.5"} still gets the full preset.

Also declares the three packages as a changesets `fixed` group. Lockstep was
incidental, not enforced: updateInternalDependencies only propagates downward to
dependents, and agent-server is a leaf, so an agent-server-only changeset bumped
only agent-server and skewed the image tag away from the npm versions. Verified
against 0.1.7, where agent-client's changelog shows it bumped solely because
agent-protocol did. Consumers pin one version for the whole stack, so that skew
would break the "does npm match the image tag?" check; the cost is a no-op bump
for packages whose code did not change.

Adds 22 tests covering file loading, default-model selection, preset precedence,
every failure mode and each removed variable, and migrates the two existing
LiteLLM tests in server.test.ts. Full suite: 155 passing.
@neuromaxer
neuromaxer merged commit 336a997 into main Aug 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant