Skip to content

feat(core,runtime,desktop): declare thinking levels on Anthropic-protocol relays - #3686

Open
qian0817 wants to merge 5 commits into
apache:mainfrom
qian0817:feat/anthropic-relay-thinking-levels
Open

feat(core,runtime,desktop): declare thinking levels on Anthropic-protocol relays#3686
qian0817 wants to merge 5 commits into
apache:mainfrom
qian0817:feat/anthropic-relay-thinking-levels

Conversation

@qian0817

@qian0817 qian0817 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

An anthropic-compatible relay fronts models the built-in metadata chain cannot resolve (no provider alias exists for it), so its models offered no thinking menu at all — while the per-model declaration mechanism that would fix it (relayModelProfiles.thinkingLevels, #2463) was gated OpenAI-relay-only.

This PR opens that gate for anthropic-compatible:

  • The gate is now per-field rather than per-provider: thinking-level declarations are legal on all three custom relays; serviceTier stays OpenAI Responses-only.
  • The declarable vocabulary is per provider: anthropic-compatible declares off|low|medium|high|xhigh|max. off joins because the protocol has a true disable wire (thinking: { type: 'disabled' }); minimal is excluded because declared levels are emitted as providerOptions.anthropic.effort, and the pinned @ai-sdk/anthropic parses that option through a closed enum (low|medium|high|xhigh|max) before any request — minimal would throw locally. The OpenAI relays keep their vocabulary unchanged (minimal|low|medium|high|xhigh|max; off there encodes reasoning_effort: 'none', which no generic relay is presumed to honor).
  • DECLARABLE_RELAY_THINKING_LEVELS becomes declarableRelayThinkingLevels(providerType); normalizeRelayModelProfiles gains an optional provider argument and stays provider-blind without one (the host-wire decode edge has no provider context — the canonical store codec has already validated fit).
  • The catalog codec asserts field-level provider fit, and the update path threads providerType through to table decode.
  • The runtime anthropic wire maps a declared off to thinking.disabled and each remaining declared tier (low|medium|high|xhigh|max) 1:1 to a passthrough providerOptions.anthropic.effort.
  • The settings page's thinking menu, bulk control, and draft seed use the per-provider vocabulary — the same controls the OpenAI relays already expose.
  • The isRelayProviderType type predicate now covers all three relay providers.

Breaking change (forward compatibility, same posture as #3309's serviceTier): older builds reject a catalog document carrying these declarations. SCHEMA_VERSION stays 1, matching the repo convention that value-range growth is not a schema change. Worth a release-note line: clear Anthropic-relay thinking declarations before downgrading.

Verification

  • packages/core: clean build + full suite pass (22/22 in the touched suites; per-provider vocabulary, codec field-level fit incl. off accept/reject on both relay families, update-path positive case)
  • packages/runtime: clean rebuild + full suite pass — 2987 pass / 0 fail (incl. wire-shape tests: declared offthinking.disabled, effort passthrough, undeclared models emit nothing)
  • packages/runtime-host: 1119 pass / 0 fail
  • apps/desktop: npm test 1340 pass / 0 fail; typecheck clean (incl. draft-seed per-provider sanitation and the bulk-control vocabulary)
  • packages/storage: 903/904 — the single failure (managed-dependency-environment-crash) reproduces on a clean checkout without these changes (environment-sensitive SQLite crash harness), verified via stash comparison
  • biome check clean on all touched files
  • Verification runs performed in a shell with rg on PATH; test runs in the repo's parallel CI-style harness without it fail on spawn rg ENOENT for pre-existing environment reasons unrelated to this change

Review focus

The trilayer consistency — vocabulary (model-thinking.ts) / codec (connection-catalog-codec.ts) / wire (model-factory.ts) — is the invariant to check: off is declarable only where the provider has a true disable wire, and only that provider's runtime branch can emit it; minimal is declarable only where the receiving SDK accepts it, so it stays OpenAI-relay-only.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: GLM 5.3 via pi — implementation, tests, and PR description, following a human-approved design (field-level gate, manual declarations only, off supported). Commits carry a Generated-by trailer.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above

@M4n5ter
M4n5ter force-pushed the feat/anthropic-relay-thinking-levels branch from 68e0f81 to 7c9a037 Compare August 26, 2026 08:57
…ocol relays

An anthropic-compatible relay fronts models the metadata chain cannot
resolve (no provider alias exists for it), so its models offered no
thinking menu at all — while the declaration mechanism that would fix
it was gated OpenAI-relay-only since apache#2463.

The gate is now per-field rather than per-provider: thinking-level
declarations are legal on all three custom relays, serviceTier stays
OpenAI Responses-only. The declarable vocabulary is per provider:
`off` joins it only for anthropic-compatible, whose protocol has a
true disable wire (`thinking: { type: 'disabled' }`); the OpenAI
relays keep their vocabulary unchanged (`off` there encodes
`reasoning_effort: 'none'`, which no generic relay is presumed to
honor).

- `DECLARABLE_RELAY_THINKING_LEVELS` becomes
  `declarableRelayThinkingLevels(providerType)`; normalize gains an
  optional provider and stays provider-blind without one (the
  host-wire decode edge has no provider context — the canonical store
  codec has already validated fit).
- The catalog codec asserts field-level provider fit, and the update
  path threads providerType through to table decode.
- The runtime anthropic wire maps a declared `off` to
  `thinking.disabled` and other tiers to a passthrough `effort`.
- The settings page's thinking menu, bulk control, and draft seed use
  the per-provider vocabulary (the same controls the OpenAI relays
  already expose).

Older builds reject a document carrying these declarations (same
forward-compat posture as apache#3309's serviceTier); SCHEMA_VERSION stays
1, matching the repo convention that value-range growth is not a
schema change.

Generated-by: Claude Sonnet 4.5 via pi
@M4n5ter
M4n5ter force-pushed the feat/anthropic-relay-thinking-levels branch from 7c9a037 to 77c3f96 Compare August 26, 2026 10:02
- packages/core/src/__tests__/model-thinking.test.ts: apply biome formatting
@Astro-Han

Copy link
Copy Markdown
Contributor

I reviewed this PR at exact head b846c8704e45d461d4fcc1f1b671263fed24bf53 (base bfba2536, 15 files +342−104, 2 commits).

Spec: NO-GO — 1×P1 + 1×P2

  • P1 — Anthropic vocabulary exposes an SDK-invalid value. packages/core/src/model-thinking.ts:71-76 and packages/runtime/src/model-factory.ts:657-667 expose minimal in the Anthropic-compatible vocabulary, then emit it as anthropic.effort. The pinned @ai-sdk/anthropic 4.0.40 accepts only low|medium|high|xhigh|max; provider-option parsing throws InvalidArgumentError before any fetch. Repro: declare or select minimal on an Anthropic relay and make a call — the newly advertised and persisted choice prevents local execution. The new runtime test covers only high and off (model-factory-thinking.test.ts:82-100), so the invalid vocabulary→wire boundary is missed. Fix by giving Anthropic relays an SDK-valid effort vocabulary (plus off) and adding a real provider-options negative/positive regression.
  • P2 — serviceTier persistence gate is too wide. The spec says serviceTier remains OpenAI Responses-only, but packages/core/src/runtime-policy/connection-catalog-codec.ts:342-355 includes both openai-compatible and openai-responses-compatible. normalizeCreateCatalogConnectionInput therefore accepts {serviceTier:'fast'} for Chat Completions relays even though supportsRelayFastServiceTier('openai-compatible', ...) is always false, leaving durable dead state. Codec tests reject it only on non-relays. Gate the check to openai-responses-compatible and add create/update/canonical negatives for Chat relays.

Positives: provider-aware create/update/draft/bulk plumbing is consistent; Anthropic off round-trips and maps to true thinking.disabled; OpenAI relays reject off; the provider-blind Host decode matches the documented trust boundary; schema-v1 posture is as designed.

Standards: 1×P3

  • P3 — docs/contracts still claim profiles are OpenAI-relay-only after Anthropic support: packages/core/src/llm-connections.ts:137-145,725-730 and nearby stale comments at packages/core/src/model-thinking.ts:310-316, apps/desktop/src/renderer/settings/provider-connection-detail.tsx:203-209, and apps/desktop/src/renderer/settings/use-connection-detail.ts:387-394. Update to custom relay / explicit OpenAI + Anthropic scope.

Other checks: git diff --check passes; worktree clean; Biome passes on the recognized changed files. The PR template, AI disclosure, substantive commit trailer, architecture, and branch/title otherwise conform. No judgment-only findings beyond the docs.

Hosted checks: windows_recovery run 33031917542 is SUCCESS; exact-head test run 33031917529 is still IN_PROGRESS, so CI is not yet terminal. Post-seal the PR is OPEN / non-draft / MERGEABLE / BLOCKED / REVIEW_REQUIRED, 0 reviews/comments, head did not drift.

What I did not check: full local suite beyond the checks noted.

Gate: exact head has P1/P2 findings and test is still in progress; it cannot be approved or merged until the Anthropic vocabulary, serviceTier gate, and docs are fixed and test turns green.


Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.

Address PR apache#3686 review (1xP1 + 1xP2 + 1xP3):

P1 — Anthropic relay vocabulary exposed an SDK-invalid value. The relay's
declared levels are emitted as providerOptions.anthropic.effort, which
@ai-sdk/anthropic 4.0.40 parses through a closed low|medium|high|xhigh|max
enum before any request; "minimal" would throw InvalidArgumentError
locally. Exclude "minimal" from the anthropic-compatible vocabulary in
declarableRelayThinkingLevels, add codec rejection + normalize-filter
tests, and extend the runtime relay test to assert every declarable level
maps 1:1 onto the SDK's effort enum.

P2 — serviceTier persistence gate was too wide. The codec accepted
serviceTier on both OpenAI relay kinds, but the read seam
(supportsRelayFastServiceTier) is Responses-only: a Chat Completions
relay declaration was durable dead state. Narrow assertProfileFieldsFitProvider
to openai-responses-compatible and add create/update/canonical negatives
for the Chat relay.

P3 — docs/contracts still claimed profiles are OpenAI-relay-only after
this PR made them legal on all three custom relays. Update the stale
wording in llm-connections.ts, model-thinking.ts,
provider-connection-detail.tsx, and use-connection-detail.ts.
@qian0817

Copy link
Copy Markdown
Contributor Author

All three findings verified against the codebase and fixed in 3ae458f (+ formatting commit d908c3488).

P1 — confirmed, fixed. Verified @ai-sdk/anthropic 4.0.40's provider-options schema is z.enum(["low","medium","high","xhigh","max"]) (dist/index.js:1129) — minimal fails parsing before any fetch. declarableRelayThinkingLevels now excludes minimal for anthropic-compatible (mirroring how off is excluded on the OpenAI wires). Since the codec's table decode and the sanitizer both derive from that function, create/update/canonical paths now reject it. Added: codec rejection test, normalize-filter test, and the runtime regression — every declarable level (low/medium/high/xhigh/max) asserted to map 1:1 onto {anthropic:{effort}}.

P2 — confirmed, fixed. Verified supportsRelayFastServiceTier returns false for everything but openai-responses-compatible, and the existing test fixture explicitly accepted serviceTier:'fast' on a Chat relay create. assertProfileFieldsFitProvider now gates serviceTier to openai-responses-compatible only (error message updated to match), with new negatives on all three write paths: create, provider-scoped update, and canonical decode for the Chat relay.

P3 — confirmed, fixed. All five stale "custom OpenAI relay" / "OpenAI-compatible relay" wordings updated to the custom-relay (OpenAI chat/responses + Anthropic protocol) scope: llm-connections.ts (×2), model-thinking.ts, provider-connection-detail.tsx, use-connection-detail.ts.

Verification: core 658/658, runtime 3038 pass / 0 fail, desktop main 1512 pass / 0 fail (8 cancelled in mcp-oauth-controller.test.js reproduce on that file in isolation pre-change), typecheck, biome format:check, and lint all clean.

…rn assertion

The focused-turn eviction test flaked in CI (run 33033873117, PR apache#3686):
after scrollTranscriptTo(bottom), turn-prompt-rail-120 never mounted
within the 10s toHaveCount window. This is the same timing family as
apache#3121, which established the jump races the virtualizer's scroll-anchor
restore: the window settles against estimated-height spacers before the
tail turn mounts, and with no further scroll event it stays settled.
The 2e3c810 guard covered only the second-half (top) jump.

Re-assert the bottom scroll once the first paint lands and dispatch the
scroll event the window recompute listens for — the same mechanism the
second half already uses. A real regression (the tail turn never
mounting at the bottom) still fails the assertion.
@github-actions github-actions Bot added the effort/L Under 1000 readable lines label Aug 27, 2026
…undtrip budget

The workhub-layout spec flaked in CI (run 33035109906, PR apache#3686): the
.workhub-result panel never became visible within the default 10s after
submitting into the WorkHub composer. The app code at that head is byte-
identical to the previous head where the test passed (the intervening
commit only touched another spec file), so this is timing, not a
regression.

The result settles through the same model roundtrip the spec's first
submit waits 20s for (line 29), plus WorkHub routing and projection
refresh on top; workhub-reconstruction.spec.ts gives the same class of
assertion 20s. Align this one with that budget instead of the 10s
default.
@Astro-Han

Copy link
Copy Markdown
Contributor

This is a synthesis of the independent blind review by @Sol-404ARE at exact head af0c9ed2bff9bb496400b96c33ebe540b0681f8a (base bfba2536, 17 files +517−131, 3 commits) compared to the sealed baseline b846c8704e45d461d4fcc1f1b671263fed24bf53. I verified the diff and the exact-head CI myself; the detailed file:line findings below are from Sol's sealed review.

What I checked myself:

  • Read gh pr diff 3686 for af0c9ed2 (thinking vocabulary per-provider, connection-catalog-codec serviceTier gate, draft/bulk UI, and the prompt-rail + WorkHub repair commits) and confirmed the titular fix is present and does not reintroduce the old minimal wire for Anthropic.
  • Checked exact-head CI: windows_recovery SUCCESS, test run 33036216987 IN_PROGRESS at seal time.
  • Checked that git diff --check and Biome pass for the changed files (per Sol's seal).

Findings from Sol's review (file:line anchored):

Spec — 1×P2 (implementation is correct; the contract text is stale)

  • P2 — PR description still describes the old vocabulary contract while the code now correctly excludes minimal for Anthropic. The PR body says off joins anthropic-compatible and other tiers pass through as effort, implying the OpenAI vocabulary plus off. The code at packages/core/src/model-thinking.ts:74-79 now correctly excludes minimal for Anthropic, the codec at connection-catalog-codec.ts:278-287 enforces that vocabulary, and runtime-policy-codec.test.ts:431-450 requires an Anthropic profile containing minimal to throw. The pinned SDK would reject anthropic.effort:'minimal' before fetch, so the secure implementation is correct — the PR contract text should be updated to off|low|medium|high|xhigh|max for Anthropic, not restored to the insecure wire.

Old findings closed:

  • Previous P1 (Anthropic minimal reaching the SDK) — closed: minimal no longer reaches the Anthropic SDK; low|medium|high|xhigh|max + off behave as expected.
  • Previous P2 (serviceTier gate too wide) — closed: serviceTier is now allowed only for openai-responses-compatible, with create/update/canonical negatives covered.
  • Previous Standards P3 (stale docs) and the two test-only repairs — closed; no coverage weakening was found.

Standards — GO (0 hard findings)

  • The four previously stale docs (llm-connections.ts, model-thinking.ts, provider-connection-detail.tsx, use-connection-detail.ts) now correctly describe custom relays as OpenAI chat/responses + Anthropic protocol.
  • Prompt-rail second paint/scroll and WorkHub 20s waits only stabilize the setup and still require the final tail turn to mount; no geometry assertion was weakened.

What I did not judge:

  • Full local test suite beyond the focused checks (the isolated worktree had no installed dependencies, so no tsc run was claimed).
  • Whether the updated PR description should also mention the exact xhigh/max mapping details for non-code reviewers.

Gate: the secured implementation itself is correct and windows_recovery is green, but the PR still has one P2 contract-text mismatch and test was IN_PROGRESS at seal, so it cannot be considered merge-ready until the PR description is aligned and test turns green.


Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/L Under 1000 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants