fix: use Copilot Auto model router and handle empty LLM responses (#91) - #253
Open
Sanjay Singh (san360) wants to merge 2 commits into
Open
Conversation
All AI features (Skill Finder, Generate Quiz, Slop or Not, etc.) picked a
model via a hardcoded family list. A family can be *selectable* yet disabled
for the caller's Copilot plan/org -- such models silently stream back an
empty response instead of failing selection, and that empty text failed JSON
parsing on every retry, surfacing as a misleading 'LLM returned invalid JSON
after 3 attempts' error for every AI feature.
selectModel() now tries Copilot's own Auto router first
(vscode.lm.selectChatModels({ id: 'auto', vendor: 'copilot' })), the same
selector VS Code's built-in Copilot Chat uses for Auto mode, so requests are
routed to a model actually enabled for the user's plan/org. It falls back to
the existing hardcoded family list for older Copilot versions.
callLlmJson() also now detects an empty streamed response as its own failure
mode (instead of letting it masquerade as a JSON parse error) and surfaces an
actionable message when every retry comes back empty.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found. |
… npm mirror zod (4.4.3->4.5.4), eslint-plugin-unicorn (73.0.0->74.0.0), @types/node (26.2.0->26.4.0), knip (6.32.2->6.33.0), and lint-staged (17.3.0->17.4.1) were all bumped by Dependabot on main within the last few hours. The internal npm registry mirror this environment is pinned to (packagefeedproxy.microsoft.io) has not yet cached these newly-published tarballs, so both 'npm install' and 'npm ci' fail with 404s for them (confirmed via 'npm view <pkg>@<version>' against the same registry). Reverts package.json/package-lock.json to the last commit before today's bump commits (c2cab82), whose versions are all confirmed available in the mirror. This unblocks local installs (and the dashboard canvas, which triggers npm install) without touching any application code. Safe to re-apply once the internal mirror catches up on these releases. Verified: npm ci, npm run typecheck, npm run lint, npm run build, npm run lockfile-lint all pass; npm test has the same single pre-existing, environment-sensitive failure as before this change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #91 —
Error: LLM returned invalid JSON after 3 attempts. Please try again.on every AI-driven feature (Skill Finder, Generate Quiz, Slop or Not, rule compilation, etc.).Root cause
All AI features go through
callLlmJson()insrc/webview/panel-llm.ts, which picked a model once viaselectModel()using a hardcoded family list (gpt-5.4-mini→gpt-5-mini→gpt-4.1-mini→gpt-4.1→ any).A model in that list can be selectable via
vscode.lm.selectChatModelsyet be disabled for the caller's Copilot plan/org. In that case it doesn't fail selection — it just silently streams back an empty response. BecausecallLlmJsonreused the same (broken) model for every retry, the empty text failed JSON parsing three times in a row, and the user saw the generic, misleadingLLM returned invalid JSON after 3 attemptserror for every AI feature, with no indication of what was actually wrong.This matches the maintainer's own diagnosis on the closed/unmerged PR #102, which suggested trying
selectChatModels({ family: 'auto' })as a simpler fix (#102 (comment)). I confirmed via VS Code's own Copilot extension source that the selector its internal "Auto" router actually uses is{ id: 'auto', vendor: 'copilot' }(notfamily: 'auto').Fix
selectModel()now triesvscode.lm.selectChatModels({ id: 'auto', vendor: 'copilot' })first (wrapped in try/catch), so Copilot routes the request to whichever model is actually enabled for the user's plan/org. Falls back to the existing hardcoded family list, then any available model, unchanged — so behavior is preserved on older Copilot versions that don't recognize theid: 'auto'selector.callLlmJson()now detects an empty streamed response as its own failure mode (instead of letting it masquerade as a JSON parse error). When every retry attempt comes back empty, it throws a clear, actionable message explaining this can happen when the model is disabled for the user's plan/org, and suggesting they pick a different default chat model. Genuine JSON parse failures keep the original message/behavior.classifyLlmFailure/buildLlmJsonFailureMessagehelpers) to keepcallLlmJsonreadable given the extra branching.CHANGELOG.mdentry.This is intentionally a small, targeted fix — it does not reintroduce the model-picker UI /
aiEngineerCoach.preferredModelsetting from the closed PR #102, per the maintainer's own follow-up steering towards the simplerauto-selector approach.Related Issues
Fixes #91
Checklist
npm run typecheckpassesnpm run lintpasses (0 errors; no new warnings introduced)npm run spellcheckpassesnpm test— all pass except a pre-existing, environment-sensitive failure insrc/core/github-app-analytics.test.tsthat also fails on a clean, unmodifiedmaincheckout (confirmed), unrelated to this changeCHANGELOG.md