feat!: load LiteLLM models from a file, and fix version lockstep - #14
Merged
Merged
Conversation
BREAKING CHANGE: the LiteLLM model list now comes only from the JSON file named
by LITELLM_MODELS_PATH. Seven env vars are removed and rejected at startup.
Configuring models meant choosing between three overlapping tiers
(LITELLM_MODELS_JSON, a LITELLM_MODELS csv, or LITELLM_DEFAULT_MODEL alone),
with four more vars supplying the per-model values the csv tier could not
express. Only the structured tier was ever used in practice, and it required
cramming a multi-kilobyte JSON document onto one env line: undiffable,
uncommentable, and awkward for a control plane to generate. The list is
structured per-model config, so it belongs in a file.
Removed, and rejected with a message naming the replacement rather than ignored:
LITELLM_MODELS_JSON, LITELLM_MODELS, LITELLM_CONTEXT_WINDOW, LITELLM_MAX_TOKENS,
LITELLM_REASONING, LITELLM_COMPAT_JSON, LITELLM_DEFAULT_THINKING. Rejecting is
the point — a stale value would leave the provider unregistered, which presents
as "the agent has no models" with nothing indicating why. An unreadable path,
malformed JSON, a non-array and an entry without an id now throw for the same
reason. parseModels drops from three tiers to one; parseCompat, modelFromId,
parseBool, parsePositiveInt and parseJsonObject are gone.
modelPreset() is unchanged and still supplies the dialect for openai/gpt-5.5 and
the DeepSeek models. It is deliberately kept: it carries thinkingLevelMap,
defaultThinkingLevel and compat.thinkingFormat, which a consumer's own model
catalogue generally does not, and DeepSeek's null entries mark levels the model
cannot do and drive clamping. File entries override presets field by field, so a
bare {"id": "openai/gpt-5.5"} still gets the full preset.
Also declares the three packages as a changesets `fixed` group. Lockstep was
incidental, not enforced: updateInternalDependencies only propagates downward to
dependents, and agent-server is a leaf, so an agent-server-only changeset bumped
only agent-server and skewed the image tag away from the npm versions. Verified
against 0.1.7, where agent-client's changelog shows it bumped solely because
agent-protocol did. Consumers pin one version for the whole stack, so that skew
would break the "does npm match the image tag?" check; the cost is a no-op bump
for packages whose code did not change.
Adds 22 tests covering file loading, default-model selection, preset precedence,
every failure mode and each removed variable, and migrates the two existing
LiteLLM tests in server.test.ts. Full suite: 155 passing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Breaking: the LiteLLM model list now comes from one place — a JSON file named by
LITELLM_MODELS_PATH.Configuring models previously meant choosing between three overlapping tiers (
LITELLM_MODELS_JSON→LITELLM_MODELScsv →LITELLM_DEFAULT_MODELalone), with four more env vars supplying the per-model values the csv tier could not express. Only the structured tier was used in practice, and it required cramming a multi-kilobyte JSON document onto a single env line — undiffable, uncommentable, and awkward for a control plane to generate from its own model catalogue.Config surface after this change:
LITELLM_BASE_URLLITELLM_API_KEYLITELLM_MODELS_PATHLITELLM_DEFAULT_MODELLITELLM_APIRemoved, and rejected at startup rather than silently ignored:
LITELLM_MODELS_JSON,LITELLM_MODELS,LITELLM_CONTEXT_WINDOW,LITELLM_MAX_TOKENS,LITELLM_REASONING,LITELLM_COMPAT_JSON,LITELLM_DEFAULT_THINKING. Each error names its replacement.Rejecting is the point. A stale value would leave the provider unregistered, which presents as "the agent has no models" with nothing indicating the cause — that is exactly how this surfaced in a real deployment. The other failure modes are loud for the same reason: an unreadable path, malformed JSON, a non-array, or an entry without an
idall throw.parseModelsgoes from three tiers to one.parseCompat,modelFromId,parseBool,parsePositiveIntandparseJsonObjectare gone.Presets are deliberately kept
modelPreset()is unchanged. It carriesthinkingLevelMap,defaultThinkingLevelandcompat.thinkingFormatforopenai/gpt-5.5and the DeepSeek models — metadata a consumer's own catalogue generally does not have. DeepSeek'snullentries mark levels the model cannot do and drive clamping, so dropping them would silently degrade the thinking selector for exactly the models we serve.File entries override presets field by field, so a bare
{"id": "openai/gpt-5.5"}still gets the full preset. Merge depth drops 3 → 2 now that provider-wide compat is a fixed baseline.Also: version lockstep was incidental, not enforced
Found while preparing this release.
updateInternalDependenciespropagates only downward, to packages that depend on the changed one:agent-serveris a leaf, so an agent-server-only changeset bumps onlyagent-server— skewing the image tag away from the npm versions. Confirmed with achangeset versiondry run (agent-server 0.2.0, others0.1.7), and by0.1.7itself: thebake-seccomp-profilechangeset named onlyagent-server, andagent-client's changelog shows it bumped solely becauseagent-protocoldid. Every release so far looked locked together by coincidence.This PR declares the three packages as a changesets
fixedgroup, so all three always share a version and the image tag matches. Consumers pin one version for the whole stack, so that skew would break the cheapest available check ("does the npm version match the image tag?"). Cost is a no-op bump for packages whose code did not change. Rationale is recorded in the root README.Verified with
fixedin place: all three →0.2.0, internal deps repinned, changelogs written.Testing
npm test— 155 passing. 22 new tests intest/litellmModels.test.tscover file loading, default-model selection, preset precedence and override, every failure mode, and each removed variable. The two existing LiteLLM tests inserver.test.tsare migrated to the file mechanism (one relied on the deleted "default model only" tier).npm run check:ci,npm run typecheck, and thegen:contractfreshness gate all clean.Consumer impact
openorange is the only consumer — verified: zero
LITELLM_*references in appx's Go control plane. It will move toLITELLM_MODELS_PATHwith a file rendered from its own model catalogue and mounted read-only, matching how it already passesMODEL_CATALOG_PATHto operator-api. That change is prepared and lands once this releases.