Skip to content

fix(models): drop the fuzzy tier from the models.dev lookup - #486

Merged
Leo310 merged 2 commits into
mainfrom
fix/drop-fuzzy-model-lookup
Sep 13, 2026
Merged

Leo310 merged 2 commits into
mainfrom
fix/drop-fuzzy-model-lookup

Conversation

@Leo310

@Leo310 Leo310 commented Sep 13, 2026 •

Copy link
Copy Markdown
Member

Fixes #480.

Problem

Model labels, context windows and capability flags come from a models.dev catalogue lookup (lookupModelInfoSync). Its last tier was a Levenshtein match over separator-stripped tokens (distance ≤ 2). One-edit tolerance is wrong for model ids, where a single character carries meaning: qwen3.8 normalizes to qwen38, one edit from the catalogue's qwen38b, so a custom-provider model was renamed to "Qwen 3 8B" and silently inherited that model's context window and vision/tool/reasoning flags. The existing digit-sequence guard was already patching one instance of this; a size suffix is the same class of problem and can't be special-cased exhaustively.

Reproduced against the live catalogue with a custom provider id:

Typed id Before After
qwen3.8 Qwen 3 8B (wrong) unchanged
qwen3:8b Qwen 3 8B Qwen 3 8B
claude-sonet-4-5 (typo) Claude Sonnet 4.5 unchanged

Change

  • Remove the Levenshtein tier. Exact, provider-prefixed and separator-insensitive (normalized equality) matching stay, so qwen3:8b / Qwen3-8B / qwen/qwen3-8b still resolve to the same entry.
  • The only lost behaviour is typo recovery, which had no real value: ids normally come from the provider's model list, and a typo'd id fails at request time anyway. Recovering the label only hid the mistake.
  • Tests: retarget the two assertions that asserted fuzzy hits, add a separator-variant case and a qwen3.8 regression case.

Not disabling the lookup for custom providers: OpenAI-compatible endpoints (Groq, Together, SAP AI Core) get correct metadata from exact and normalized matches, and the cross-provider precedence tests cover exactly that.

Checks

bun run check, bun run format, bun run lint, bun run test (1675 passed).

AI assistance: written with AI coding agents from the maintainer's brief; reviewed and tested by the maintainer.

The catalogue lookup ended with a Levenshtein match on separator-stripped
tokens. One-edit tolerance is wrong for model ids, where a single character
carries meaning: "qwen3.8" normalizes to "qwen38", one edit from
"qwen38b", so a custom-provider model was renamed to "Qwen 3 8B" and
inherited that model's context window and capability flags.

Keep exact, prefixed and separator-insensitive matching ("qwen3:8b" still
resolves to "qwen3-8b"); require equality after normalization.
@greptile-apps

greptile-apps Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

Removes edit-distance matching from the models.dev catalogue lookup so near-miss model identifiers cannot silently inherit another model’s label, context window, or capabilities.

  • Preserves exact, provider-prefixed, and separator-insensitive equality matching.
  • Adds regression coverage for qwen3.8, separator variants, and typo rejection.

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness, security, or repository-rule issues identified.

The lookup now requires exact or separator-normalized equality, preventing near-miss identifiers from inheriting incorrect catalogue metadata while preserving the documented valid matching paths.

Important Files Changed

Filename Overview
src/providers/modelsDevApi.ts Removes fuzzy model matching while retaining exact and normalized identifier matching.
test/providers/modelsDevLookup.test.ts Updates lookup expectations and adds coverage for separator equivalence and one-edit mismatches.

Reviews (2): Last reviewed commit: "Merge branch 'main' into fix/drop-fuzzy-..." | Re-trigger Greptile

@Leo310
Leo310 merged commit 240a5fe into main Sep 13, 2026
3 checks passed
@Leo310
Leo310 deleted the fix/drop-fuzzy-model-lookup branch September 13, 2026 13:46
Leo310 added a commit that referenced this pull request Sep 14, 2026
Stability release for the Obsidian 1.13 installer (Electron 43 / Node 24):
the renderer crash on the first chat send (#492) and the transport
connection error (#501), plus the streaming render coalescing (#484), the
checkpoint callbacks strip (#483), and the indexing fixes since 2.1.0
(#486, #487, #490, #491, #493, #495, #497, #498, #499, #500).

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Model name formatting corrupts names (e.g., "qwen3.8" becomes "Qwen 3 8b")

1 participant