Skip to content

[ENHANCEMENT] Dynamic Thinking Effort: AI self-decides per-turn reasoning effort (experimental) — design + implementation plan #28

Description

@easonLiangWorldedtech

[ENHANCEMENT] Dynamic Thinking Effort — letting the AI decide its per-turn reasoning effort (experimental)

Design + implementation plan. Research-backed (2026-08 industry + arXiv review).
Related upstream gap: Roo Code issue #7048 (effort override semantics).

1. Problem and goals

Today reasoning effort can only be fixed once in Settings (static per-profile). It is not possible to:

  • Let the model decide its effort based on task complexity
  • Raise/lower the effort mid-task as needed (e.g. start cheap, escalate to high only when tests fail)
  • Let the user adjust it in chat instantly, without going to Settings

Goals (scope of this issue):

# Requirement
F1 Fix: Anthropic adaptive models receive no thinking content + no thinking-token data + no effort envelope (see §4)
F2 An experimental settings toggle that enables the whole feature
F3 Native tool set_thinking_effort: the AI decides its effort mid-task, with a display in the chat (same pattern as switch_mode)
F4 Top dashboard (TaskHeader) shows the current effort, including the init state (the resolved default at task start)
F5 Bottom manual button (composer bottom bar): temporary, current chat only, exactly the same scope as the AI's tool use (the same task-local state)
F6 Orchestrator: new_task supports a thinking-effort input (model-specified); the user can switch it before entering the subtask (before clicking the button)

Non-goals (backlog): per-message quick chips / prompt trigger words ("ultrathink"), session-level thinking ledger / overthinking warnings, per-mode effort policies, cost-saver mode.

2. Permission question: does the AI's auto-switch need approval?

Conclusion: no. set_thinking_effort executes directly (no approval gate) and is bounded by guardrails instead:

Reason Explanation
Non-destructive It changes no files, runs no commands, and does not affect the permission context. Compare: switch_mode changes the mode (which changes the available tools/permissions), so it needs approval; new_task spawns a new task, so it needs approval. Effort is neither.
Reversible It can be changed back at any time (by the user or the model); the bottom button is the undo mechanism.
Bounded cost Effort is clamped to the levels the model supports, so it cannot escalate unbounded; plus an escalation cap (guardrails below)
Product logic Requiring human approval for every switch defeats the purpose of the dynamic loop (the model could not self-adjust while the user is away) and adds a latency round-trip each time. Industry precedent (Claude Code /effort + adaptive default, OpenCode adaptive-thinking plugin) also has no approval.

Guardrails (in place of approval):

  1. Always notify: every change by the model or the user emits a display line in the chat stream (never a silent change)
  2. Escalation cap: at most N upward adjustments per task (3 suggested); beyond that, clamp + notify
  3. Oscillation detection: low → high → low within 3 steps notifies once (possible cost spiral)
  4. Hard cap: never exceeds the highest level the model supports (per the supportsReasoningEffort capability array)
  5. (Fallback, off by default) a "effort changes require approval" setting could be added for conservative users — out of scope for v1

3. High-level UI/UX design

3.1 Single source of truth: task-level effective effort

resolution order (strongest first):
  1. task.runtimeThinkingEffort   ← written by the tool or the bottom button (same state, last-write-wins)
  2. apiConfiguration.reasoningEffort   ← the Settings persisted value
  3. model.reasoningEffort default  ← the model definition
  • runtimeThinkingEffort is task-layer: stored with the task (persisted in its history item) and restored when the task is reopened from history; never written to Settings, and never pollutes the profile. F5's "same scope as the AI's tool use" = both sides read/write this same field.
  • Every display surface shows the effective value + source (default / Zoo (auto) / you), so the user can always tell where the value came from — this is what makes "let the AI decide" trustworthy (transparency builds trust).

3.2 The surfaces

(a) Experimental settings toggle

  • Follows the existing ExperimentalSettings.tsx generic pattern: EXPERIMENT_IDS.DYNAMIC_THINKING_EFFORT + i18n (settings:experimental.DYNAMIC_THINKING_EFFORT.name/.description)
  • Toggle semantics: "Let the model decide its thinking effort per step, and let you adjust it in-chat"
  • Off = the set_thinking_effort tool is not exposed, so model-driven mid-task switching is disabled. The composer toggle and TaskHeader chip are capability-gated (they render whenever the selected model advertises per-request effort support — registry capability or an F7 declaration) and stay available regardless of this toggle

(b) Top dashboard (TaskHeader) chip

  • Placement: near the cost/tokens row, a small chip:
┌──────────────────────────────────────────────┐
│ Fix the login bug…        🧠 High · Zoo      │  ← thinking chip (new)
│ $0.05 · 2.1k in · 980 out · 42%             │
└──────────────────────────────────────────────┘
  • Init state: at task start it shows the default resolved by the resolution order (e.g. 🧠 Medium · default); a reopened task shows its restored value with the saved source — the user does not have to wait for the first switch to know the current level
  • The source badge updates with changes: · default· Zoo (model tool) → · you (manual)
  • Tooltip (shipped): "Thinking effort: {{effort}} ({{source}})" — e.g. "Thinking effort: High (you)"; source labels: default / Zoo (auto) / you; adaptive-class models additionally show "This model decides its effort automatically — your selection is soft guidance only."
  • Provider honesty: for providers without native adaptive, the tooltip truthfully states that effort is a per-request parameter (takes effect on the next request), no "adaptive" overselling

(c) Bottom manual button (ChatTextArea bottom bar)

  • Placement: between the API-profile selector and the auto-approve (⚡) control, small button in the same row (same border/hover treatment as the sibling selectors):
[ Code ▾ ]  [ api-profile ▾ ]  [ 🧠 High ▾ ]  [⚡▾]   [ Send ]
  • Click → small menu: effort levels (only values the model's supportsReasoningEffort supports, following the existing ThinkingBudget clamp logic), with an optional "Reset to default" at the top
  • Selection applies immediately: writes task.runtimeThinkingEffort + rebuilds the handler + posts to the webview (chip updates), and the chat emits a display line (source = you)
  • Persistence semantics: task-layer — the selected value is stored with the task and restored when the task is reopened from history; the Settings persisted value is never written to
  • Visibility: hidden (renders nothing) unless the selected model advertises per-request effort support (registry capability, or an F7 declaration) — no separate disabled state in v1

(d) Orchestrator (new_task) pre-enter effort

  • Add an optional thinking_effort to the new_task tool schema: the model can specify a starting effort for a subtask (e.g. "this subtask is mechanical, use low")
  • Add an effort selector to the ChatRow newTask ask block (currently has mode + message + todos + approve/enter buttons):
┌─ New task ─────────────────────────────────────┐
│ Mode: Code                                      │
│ Message: Implement the retry logic…             │
│ Thinking: [ High ▾ ]   ← prefill = model-specified value
│                                    [ Enter ]    │
└─────────────────────────────────────────────────┘
  • The user can change it before clicking Enter; after entering, this value becomes the child task's initial effective effort (the child's TaskHeader chip init state shows it directly)
  • Prefill when the model specified nothing = the parent's current effective effort

(e) Chat stream display (same as switch_mode)

🧠 Zoo raised thinking to High — "multi-file refactor, 3 modules affected"
  • Partial streaming follows the switchModeTool.handlePartial pattern (task.ask("tool", ...))
  • Manual user changes use the same say type with a different source (slight icon/text adjustment), for task-history auditability

3.3 Wireframe overview

┌─ VS Code Panel ──────────────────────────────────────────┐
│ ┌ TASK HEADER ────────────────────────────────────────┐  │
│ │ Task title…            $0.05 · tokens …   🧠 H·Zoo │  │ ← (b)
│ └────────────────────────────────────────────────────┘  │
│                                                          │
│  👤 refactor the auth module                             │
│                                                          │
│  🧠 Zoo set thinking to Medium — "scoped, low risk"     │ ← (e)
│                                                          │
│  🤖 …working…                                            │
│                                                          │
│  🧠 New task (Code) · Thinking: [High ▾]     [Enter]    │ ← (d)
│                                                          │
│ ┌ COMPOSER ──────────────────────────────────────────┐   │
│ │ [Code ▾] [profile ▾] [🧠 High ▾] [⚡▾]       [➤]  │   │ ← (c)
│ └────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────┘
Settings → Experimental → ☑ Dynamic thinking effort        ← (a)

3.4 Design decision summary (mapped to research findings)

Decision Rationale
Controls in three layers: Settings (persistent) + composer bottom bar (temp) + dashboard (display) Follows the 2026 industry convergence direction: VS Code/Cursor use a picker (≈ our bottom bar), Cline/Roo use settings; the three layers do not conflict — settings = persistent default, bottom bar = temp for this chat, tool = temp by the AI
"Auto" = the experimental feature itself (not an option inside the selector) This repo's effort already has a per-model default; "the model decides" only becomes complete with the tool + adaptive pass-through, so an experimental toggle is the clean boundary that does not fight the existing settings semantics
No approval, always notify §2
Source badges (default/Zoo/you) transparency → trust; industry patterns (VS Code hover shows the model, Cursor picker labels) all do state visibility
Only list the levels the model supports follows the supportsReasoningEffort capability array (OpenAI gpt-5.2 default none, o1 only low/med/high, etc.), avoiding sending a level that 400s
Anthropic uses native adaptive + soft envelope; the rest use a per-request parameter §5 provider matrix; UI copy is truthful (no overselling)

4. Fix part (Anthropic adaptive models)

Code evidence (SDK 0.109.1 already supports every required type — no dependency upgrade needed):

  1. Thinking content is never received: src/api/providers/anthropic.ts never sends the display parameter; Opus 4.7 / Fable 5 / Mythos class default to display: "omitted" → the stream handler has a thinking_delta case but receives nothing. Fix: the adaptive branch of getAnthropicProviderReasoning (src/api/transform/reasoning.ts) emits { type: "adaptive", display: "summarized" }
  2. No thinking-token data: usage.thinking_tokens is not parsed. Fix: add reasoningTokens to the message_start / message_delta usage chunks (0 ⇒ the model skipped thinking that turn — exactly the "how much did the model actually think" telemetry)
  3. No effort envelope: output_config.effort (low/medium/high/xhigh/max, soft guidance) is a separate top-level field (putting it inside thinking raises a ValidationException). Fix: the transform returns { thinking, outputConfig? }, and both requestParams branches in the handler merge output_config
  4. The Bedrock handler (Converse API) gets the same fix (additionalModelRequestFields)

These three are standalone fixes on their own (Opus 4.7 users seeing no thinking at all is already a bug) and can ship ahead of the feature.

5. Provider support matrix

Provider Native "model decides" Tool-driven (F3) Notes
Anthropic 4.7+/Fable 5/Opus 5 ✅ adaptive ✅ adaptive + output_config.effort envelope In adaptive mode the API has no per-turn effort parameter; interleaved thinking happens automatically between tool calls; Opus 4.7 must be explicitly sent adaptive, otherwise thinking is off
Gemini 3.x / 2.5 ✅ dynamic (default) thinkingLevel per-request 3.1 Pro cannot be fully disabled; thoughtSignature already handled
OpenAI native / OpenRouter / OpenAI-compatible ⭕ none (omit = model default) reasoning_effort per-request Clamped to the capability array; gpt-5.2 default none
DeepSeek V4 ⭕ (thinking on + default high) ✅ (coarse-grained: medium/xhigh silently map to high) reasoning_content round-trip already handled (openai-format.ts)

6. Implementation plan (file-level)

Phase 0 — Fix (independently mergeable, ~0.5d)

  • src/api/transform/reasoning.ts: new getAnthropicProviderReasoning return shape + ADAPTIVE_THINKING_EFFORT_LEVELS
  • src/api/providers/anthropic.ts / bedrock.ts / anthropic-vertex.ts: output_config merge + thinking_tokens parse
  • Unit tests: src/api/transform/__tests__/reasoning.spec.ts (adaptive + display + outputConfig)

Phase 1 — Experimental toggle + types (~0.5d)

  • packages/types/src/experiment.ts: dynamicThinkingEffort
  • src/shared/experiments.ts: DYNAMIC_THINKING_EFFORT config
  • webview-ui/src/components/settings/ExperimentalSettings.tsx + i18n (en + zh-TW, other locales follow)

Phase 2 — Tool + Task state (~2d)

  • packages/types/src/tool.ts: set_thinking_effort ToolName
  • src/shared/tools.ts: ALWAYS_AVAILABLE_TOOLS + NativeToolArgs + toolParamNames (thinking_effort)
  • src/core/prompts/tools/native-tools/set_thinking_effort.ts: schema (effort enum, reason required) + guidance copy ("pick the lowest safe effort; escalate only for ambiguity/debugging/risky changes; state your reason")
  • src/core/prompts/tools/native-tools/index.ts: registration
  • src/core/prompts/tools/filter-tools-for-mode.ts: gating = experiment on AND model supports per-request effort (per the generate_image / run_slash_command precedent; prompt-cache rule: once a task starts, the tool list is stable and never grows/shrinks with state)
  • src/core/tools/SetThinkingEffortTool.ts: executor (no approval); clamp against the capability; escalation cap + oscillation detection; write task state; say display
  • src/core/assistant-message/presentAssistantMessage.ts + NativeToolCallParser.ts: dispatch case (following switch_mode)
  • src/core/task/Task.ts: runtimeThinkingEffort + setRuntimeThinkingEffort(effort, source) (merge into the apiConfiguration copy + updateApiConfiguration() rebuilds the handler — existing profile-switch precedent; the value is written into the task's history entry, so it survives closing and reopening) + post to the webview

Phase 3 — Webview UI (~2d)

  • webview-ui/src/components/chat/TaskHeader.tsx: chip + source badge + init state
  • webview-ui/src/components/chat/ChatTextArea.tsx: ThinkingEffortButton (bottom bar)
  • webview-ui/src/components/chat/ChatRow.tsx: case "setThinkingEffort" display (ask/say both states)
  • ExtensionState / message handler plumbing (extension→webview effort change events)
  • Vitest: ChatRow case, chip, button interactions (webview-ui AGENTS.md two-layer strategy)

Phase 4 — Orchestrator (~1d)

  • src/core/prompts/tools/native-tools/new_task.ts: + optional thinking_effort
  • src/core/tools/NewTaskTool.ts: ask JSON carries the effort; pass-through
  • webview-ui/src/components/chat/ChatRow.tsx newTask block: effort selector (changeable before Enter)
  • src/core/webview/ClineProvider.ts: delegateParentAndOpenChild accepts thinkingEffort? → set at child task init

Order

P0 → P1 → P2 → P3 → P4; P0 can proceed independently. Total estimate ~6d (single developer).

7. Risks

Risk Mitigation
Cost spiral (model over-escalates / oscillates) §2 guardrails: cap + detection + always notify + user can lower it instantly
Prompt-cache bust (conditional tool injection) The tool is only exposed when experiment on + model supports it, and is stable within a task; the description is static
Anthropic granularity is coarse (adaptive is on/off + soft envelope only) UI copy is truthful; for it, a tool change is the envelope, not a hard level
Old o1-generation thinking + tool-call limitations Gated by the model capability flags; the tool is not exposed when unsupported
Users assume "temp" persists Copy is explicit: the value is stored with the task and restored when the task is reopened; the Settings value never changes

8. Test plan (per AGENTS.md test pyramid)

9. Acceptance criteria

  • Experiment on + supported model: the AI can call set_thinking_effort mid-task, the chat emits a display line, and the next request actually carries the new effort (verifiable via network/logs)
  • Top dashboard chip: shows the default as soon as the task starts; updates immediately after a model or user change + correct source
  • Bottom button: affects only the current task; the selected value is stored with the task and restored when the task is reopened from history; the Settings value is never written to
  • Opus 4.7 / Fable class: thinking text visible (display fix) + thinking-token data present (usage fix)
  • Orchestrator: newTask can carry an effort; changeable before Enter; effective at child init
  • Experiment off: no set_thinking_effort tool exposed (the model cannot switch mid-task); the composer toggle / chip are unaffected (capability-gated)
  • No approval gate; every change is notified; the escalation cap holds

Execution status (synced 2026-08-26)

Synced 2026-08-26: trial composite Zoo-Code-Org#1379 (whole series + e2e addenda + F7) pushed at 27a2e97df — all CI green (Code QA 8 jobs incl. Build test VSIX, E2E Tests (Mocked), Release Validation), CodeRabbit clean (all findings fixed, threads resolved), patch coverage 97.31% (check codecov/patchSuccessful, target 80% — pass) / webview patch 97.67% (check codecov/patch/webview-patch, target 70% — pass). Docs: Zoo-Code-Docs#48 open, Docusaurus Build Check green. Remaining (maintainer-only): merge Zoo-Code-Org#1379 + Zoo-Code-Org#48.

Base reference: upstream/main = 87077e1b1 (moved 2026-08-23: 11 commits since db52d7f incl. v3.80.0 release prep Zoo-Code-Org#1347, Zoo-Code-Org#1351/Zoo-Code-Org#1340 async fixes, Zoo-Code-Org#1323 task persistence; merge-tree preview dte-3+newmain=4be3fe72, dte-5+newmain=b534d097 = clean). ⚠️ Zoo-Code-Org#1345 (ViX3L feat(ollama): reasoning effort selectors, OPEN non-draft, 15pass/2fail codecov/patch+e2e-mock, REVIEW_REQUIRED, unmerged) adds webview ReasoningEffortSelector into ChatTextArea bottom bar (PR-4's specified toggle slot) + utils/reasoning-effort.ts + SettingsView refactors → PR-4 must base-refresh post-Zoo-Code-Org#1345 and place the toggle adjacent; PR-3/PR-5 zero overlap. (prior: db52d7f = Gemini Flash-Lite Zoo-Code-Org#1334 merged 2026-08-22.) PR-1/PR-2 base = 1ad8f528d (1 commit behind, gemini-only, non-conflicting → no rebase; re-triggering CI/CodeRabbit not worth it). Wave-2 (PR-3/PR-5) will base-refresh via git merge upstream/main at launch so they are truly on the latest main (gemini commit cancels out of the main..head diff as a common ancestor).

F1 fix — delivered

Feature rollout — 5 stacked PRs (one issue each, upstream):

PR Issue Branch Base State
1/5 experimental setting Zoo-Code-Org#1328PR Zoo-Code-Org/Zoo-Code#1336 (a05830c + 1cf4f0d + base-merge 5db5cf4) feat/dte-1-experiment main DONE (final, on latest main): merge 5db5cf4 = 1ad8f52/Zoo-Code-Org#1069 (clean ort, locale coexistence re-verified); new head 15/15 CI green; CodeRabbit real review new range = 'No actionable comments 🎉'; Codecov all-coverable-lines; final comment updated in place (base-update section); awaiting maintainer merge (BLOCKED = review policy)
2/5 task state + metadata + envelope Zoo-Code-Org#1329PR Zoo-Code-Org/Zoo-Code#1338 (6ea45b3 + base-merge 9275aa1 + fix 14d1f35 + docstring 90b47b0, UNDRAFTED) feat/dte-2-task-state main 🔄 MAJOR RESOLVED (noAct 🎉) + Description PASS + 100% patch cov (30/30+10/10). Premature FINAL 09:11 REJECTED (Docstring 33%<80%) → JSDoc instruction → docstring commit 90b47b0 pushed 09:38 (all 3 modified fns documented). CodeRabbit settled 01:50:35Z re-scanned 90b47b0 but docstring still 33.33% = STALE CACHE (verified: all 7 diff fns documented in code = 100%; bot noAct 🎉 + check pass = non-blocking). 09:54 override: stop docstring iteration, finalize. **10:22 agent stuck on final PATCH (~6.5min zero I/O) → interrupt + finish instruction; ALL substantive gates VERIFIED (14/14 + noAct 🎉 + 100% cov) → final comment PATCH landed 03:02:37Z (5345 chars; live-verified: 90b47b0 + 14/14 + stale-cache note + corrected title/body line + signature). FINAL report received 11:0x (all gates self-verified, hard-rule compliant). ✅ COMPLETE (awaiting maintainer merge). Docstring-stale root cause (agent closing report 11:0x): CodeRabbit's incremental review skipped both JSDoc files as "similar to previous changes (2)" so the coverage check never re-measured; the /meow full re-review (posted 02:18:55Z) never produced a fresh walkthrough (~9h later still the 01:50:35Z run) — stale heuristic proven, non-blocking (noAct + check pass).
3/5 set_thinking_effort tool Zoo-Code-Org#1330 feat/dte-3-native-tool PR1+PR2 🚀 LAUNCHED 10:22 (wave 2): wt-dte-3; fcc3cf4 feat commit (33 files +1440; SIZE approved by user — do not grow) + 0ab4a60 base-refresh to main 78c712a; DRAFT PR Zoo-Code-Org/Zoo-Code#1354 OPEN (review fixes 19954d3 landed: 24 files +266/−75 — parser strictness, 'disable'-only capability hidden, baseline-seeded oscillation guard, i18n values 17 locales, 2 out-of-scope replies → PR-1 Zoo-Code-Org#1336; parent-verified 19/19 checks green) → bot tail verified by parent 22:3x: all 6 findings confirmed/withdrawn (11:49-11:50Z) = CLEANuser directive 22:3x: e2e moves to SEPARATE addendum PR-6 (feat/dte-3-e2e from 19954d3) — undraft NOW (gates already met: 19/19 + bot clean + 100% cov) → agent e06e4e01 executing: undraft Zoo-Code-Org#1354 → addendum PR-6 DRAFT → CI → undraft → bot → final
4/5 webview UI Zoo-Code-Org#1331 feat/dte-4-webview-ui PR3 🚀 LAUNCHED (user directive 16:1x): wt-dte-4 @ 0ab4a60 (stack dte-3 head); launch re-verify done: Zoo-Code-Org#1345 still OPEN non-draft (base 78c712a, 17pass/2fail, MERGEABLE) → pending-overlap rule (note in PR body; toggle lands adjacent to their selector post-merge) + per-tab Zoo-Code-Org#977/Zoo-Code-Org#981 NOT merged → extension-state push path + follow-up note; pnpm install done; agent: implement (TaskHeader chip + composer toggle + display + message types) → JSDOM + Playwright CT + e2e Part C round-trip → DRAFT → CI → undraft → bot → 100% cov
5/5 orchestrator new_task Zoo-Code-Org#1332 feat/dte-5-orchestrator PR2 🚀 LAUNCHED 10:22 (wave 2): wt-dte-5; DRAFT PR Zoo-Code-Org/Zoo-Code#1355 OPEN @ f6410bb (feat 146c5c8 = 15 files +1041/−16, SIZE approved by user — do not grow; base-refresh to main 78c712a clean; all post-merge verification green; also fixed ChatView approval pass-through gap found in testing) → e2e files written uncommitted (16KB test + subtasks DTE markers) → user directive 22:3x: e2e moves to SEPARATE addendum PR-7 (feat/dte-5-e2e from f6410bb)parent verified 3 UNRESOLVED MAJOR CodeRabbit comments (NewTaskTool boolean capability / ClineProvider post-mode-switch child revalidation / ChatView prefill normalization — full fix specs extracted + bot's proposed ChatView diff) → agent 4c597817 executing: STEP 1 e2e → addendum branch (DRAFT PR-7) → STEP 2 fix 3 Majors (1 fix commit) → STEP 3-5 push → fresh CI green → reply to bot → THEN undraft → cov → final

Per-PR protocol: draft PR until CI fully green → undraft → CodeRabbit review fully addressed → 100% test coverage on patch lines → only then next PR. Each PR <= 1000 diff lines (standalone, vs stacked base). Max 2 implementation agents concurrent; waves [1∥2] → [3∥5] → [4].

Design updates since original post:

  • Composer button placement: between the API-profile selector and the auto-approve (⚡) control (user-visible row order, plan §3.2(c)/§3.3 wireframe).
  • §3.5 reuse design: effort body fields stay in the transform layer (RequestConfigBuilder = SDK options only); per-request override rides metadata.reasoningEffort (same pattern as abort signal / Bedrock metadata.thinking); webview effort display state goes in the per-tab view state container (sequencing dependency on the view-local-state merge noted).
  • §8/§9 updated for the webview-ui two-layer test strategy (Vitest+JSDOM behavioral + Playwright CT visual snapshots in the same PR) and Codecov gates (80% src patch / 70% webview patch; requirement: 100% on patch lines).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions