Add per-role effort with per-backend translation - #54
Conversation
Closes the gap reported in AMAP-ML#38: the harness discovered models' supported reasoning efforts but offered no way to set one. The parameter is named "effort", following Anthropic's umbrella term: on Claude it is a distinct control from thinking, and on backends whose only dial is reasoning-specific it drives that dial. One normalized scale - min, low, med, high, xhigh, max: the union of the levels the backends document - selectable globally (--effort / [run] effort) and per role (--manager-effort, [run.roles.auditor] effort, ...), resolving down the same fallback chain as --<role>-agent/--<role>-model. Unlike models, the scale is backend-independent, so it safely crosses an explicit agent boundary. Each backend translates a level into its documented dial, and a level a backend lacks maps to its nearest supported one: - codex: `-c model_reasoning_effort="<level>"` (min and med spell out to minimal and medium; max -> xhigh); - claude_code: the CLAUDE_CODE_EFFORT_LEVEL environment variable (min -> low, med spells out to medium, rest native; the env var also outranks a session /effort choice, which is the isolation a harness role needs); - opencode: `--variant <name>` - OpenAI-style models ship variants named after the effort levels themselves (min/med spell out to minimal/medium), other providers ship a subset and accept custom variants from opencode.jsonc, so the requested name passes through as-is; - deepseek_harness: `reasoningEffort` in the dsh profile patch the runner already writes (low|high|max natively; DeepSeek's own API docs map medium and xhigh onto high; min -> low). Every episode's metadata records both effort (requested) and effort_effective (after the backend mapping) so the requested depth and any substitution stay auditable. The Web API accepts roles.<role>.effort on run creation, validates it at the boundary, and forwards it to the worker; the value rides along in owner.json role_configs for resume. tests: config keys (valid/invalid, per-role), CLI fallback chain, the full translation table per backend including nearest-level substitution, the dsh runner patch contents, opencode wiring, episode metadata recording, supervisor validation and worker-command forwarding.
|
@lerogo Saw you self-assigned #38 — this PR implements it (per-role @lerogo 看到你认领了 #38——这个 PR 实现了该功能（按角色的 |
The other half of AMAP-ML#38: the web UI now both sets and shows the effort level. - Run creation: each role card gets an Effort select next to Harness and Model, fed by the new `defaults.effort_levels` list in /api/meta (falls back to the built-in scale on older servers). "Default" leaves the choice to the worker's config chain. - Run details: a role-binding strip above the tabs shows each public role's model with its effort as a badge, sourced from the run's durable owner role_configs. - The snapshot provenance projection (_safe_role_configs) passes a valid `effort` through instead of stripping it, so the strip works for historical runs too; invalid values are still dropped. tests: provenance keeps a valid role effort and drops an unknown one; frontend core suite and tsc/vite build pass; verified visually in the running workbench (picker in all three role cards, chips on a created run).
|
Pushed a second commit covering the other half of #38 — the web UI now sets and shows effort:
Verified in the running workbench (picker renders in all three role cards; a run created with 第二个 commit 补上 #38 的另一半——Web 界面现在既能设置也能展示 effort：创建任务时每个角色卡新增 Effort 下拉框；任务详情顶部新增角色配置条，以徽标形式显示各角色的 effort（数据来自 |
|
No need to do mapping, just use the corresponding depth of thought of the model directly. Could you help modify it and then test it more in practical use? Thank you very much. |
Review feedback on AMAP-ML#54: no normalized scale, no nearest-level substitution - a role's effort is the model's own depth-of-thought value, passed through untouched. - Each adapter accepts exactly the levels its backend documents (codex: minimal|low|medium|high|xhigh; claude_code: low|medium|high|xhigh|max; deepseek_harness: low|high|max) and rejects anything else with a clear error naming the valid set. OpenCode keeps free-form variant names, since variants are per-model presets that may be user-defined in opencode.jsonc. - BACKEND_EFFORT_LEVELS in types.py is the single source of those sets; the CLI/config keep only a union-level typo guard because the backend is not known at parse time - the adapter (or, for Web runs, the supervisor boundary, which validates against the role's own backend) makes the final call. - effort_effective is gone from episode metadata: with no mapping the recorded `effort` is exactly what the backend received. - The web workbench effort picker now offers the selected model's own discovered levels (model catalogue `reasoning_efforts`), falling back to the backend's documented set; changing the backend resets the choice since the sets differ. Practically tested with a real claude_code run (--effort low, 3-role MEA loop to completion): all 10 episodes carry CLAUDE_CODE_EFFORT_LEVEL=low in the spawned commands and "effort": "low" in metadata; an unsupported level (--effort minimal on claude_code) fails at agent setup with the level list in the message.
|
@lerogo Done — mapping removed in bc6f116; the effort value is now the model's own depth-of-thought level, passed to the backend verbatim. Each backend accepts exactly the levels it documents and rejects anything else with a clear error naming the valid set ( Practically tested with a real @lerogo 已按建议修改（bc6f116）：去掉了映射，effort 取值即模型自身的思考深度等级，原样传给后端；每个后端只接受自己文档中的等级，其余直接报错并列出可用值（ 已实测：用真实 |
|
@lerogo Following up on "test it more in practical use" — I have now also run the two remaining backends end-to-end against the real DeepSeek API, on their current releases: DeepSeek Harness ( OpenCode (1.18.19, With the earlier claude_code run, three backends are now verified end-to-end in practice (claude_code, deepseek_harness, opencode), each receiving the effort value verbatim and failing fast on levels it does not document; codex uses the same pass-through path and is covered by the unit suite. @lerogo 继续跟进"在实际使用中多做测试"——我已在真实 DeepSeek API 上，用当前版本把其余两个后端也做了端到端测试： DeepSeek Harness（ OpenCode（1.18.19， 加上此前的 claude_code 运行，已有三个后端完成端到端实测（claude_code、deepseek_harness、opencode）：effort 值均原样透传，并对各自文档之外的级别快速报错；codex 走同一条透传路径，由单元测试覆盖。 |
The CLI and config validated effort against a hard-coded union of the built-in backends' levels, while the web supervisor validates against the role's own backend and accepts any well-formed name for OpenCode. That split rejected legitimate OpenCode variant names (user-defined in opencode.jsonc) from the command line and the config file only. - Drop the argparse `choices` guard from --effort/--<role>-effort and the union check from config.toml parsing; both now accept any non-empty name up to 64 characters without NUL - the same shape check the supervisor applies. - Remove EFFORT_CHOICES from types.py: BACKEND_EFFORT_LEVELS is the only remaining source, consulted by the adapters and the supervisor at the role->backend boundary, so CLI, config, and web now behave identically - the value travels untouched to the role's backend and is judged only there. - README (en/zh-CN): note that any well-formed name is accepted at the boundary and validated by the backend it is addressed to. Practically tested: a custom OpenCode variant (--effort deep, defined in the workspace opencode.jsonc) runs from the CLI through a full MEA loop to completion - all 14 episodes carry --variant deep and "effort": "deep" - while --agent claude_code --effort deep still fails at agent setup naming the valid levels.
|
One more refinement surfaced by the practical testing, pushed as 46e1de0: the CLI and config file used to validate All three entry points now behave identically: the value travels untouched to the role → backend boundary and is judged only there, by the backend it is addressed to. 实际测试还暴露出一处需要打磨的地方，已随 46e1de0 推送：此前 CLI 和配置文件用内置后端等级的硬编码并集来校验 现在三个入口行为完全一致：值原样传递到"角色 → 后端"边界，只在那里由目标后端做判断。 |
|
Thank you for your PR. I referred to your code to implement this part, but since I have made many local changes, this PR will not be merged. Thank you very much for your participation. Additionally, we have released a new version. Could you help us test it? #56 |
|
Totally understood — the feature landing in v0.1.7 is what matters, and I'm glad the PR could serve as a reference. Closing this one. I'll take you up on testing the new release and will report anything I find on #56's features. Thanks for the reviews along the way! 完全理解——功能在 v0.1.7 落地才是最重要的，很高兴这个 PR 能作为参考。关闭此 PR。我会接下新版本的测试，#56 相关功能如有发现会及时反馈。感谢一路以来的 review！ |
|
@lerogo Done — I tested the v0.1.7 reasoning-effort feature end-to-end on claude_code, opencode, and deepseek_harness against real providers; the full report (what works, two observations, and an offer to cover dsh's @lerogo 已完成——我在 claude_code、opencode 和 deepseek_harness 三个后端上对接真实服务，端到端测试了 v0.1.7 的思考强度功能；完整报告（正常部分、两个观察、以及为 dsh 补上 |
2 participants
Closes #38.
The harness discovered models' supported reasoning efforts (
model_catalog) but offered no way to set one. This PR adds aneffortsetting whose value is the model's own depth-of-thought level, passed to the backend verbatim — no cross-backend mapping or substitution.Usage
Globally, per role, or per run, resolving down the same fallback chain as
--<role>-agent/--<role>-model:lh-harness run --task "..." --effort medium --auditor-effort lowThe generated config ships
effort = "medium"in[run]and a commented# effort = "medium"in every role section.Per-backend dials
Each backend accepts exactly the levels it documents and rejects anything else with a clear error naming the valid set:
-c model_reasoning_effort=minimal,low,medium,high,xhighCLAUDE_CODE_EFFORT_LEVELenv (outranks a session's/effort)low,medium,high,xhigh,maxreasoningEffortin the dsh profile patch the runner already writeslow,high,max--variantopencode.jsoncpass throughBACKEND_EFFORT_LEVELSintypes.pyis the single source of those sets. Every entry point behaves the same way: the CLI, the config file, and the Web API all accept any well-formed name and the value is judged once, at the role → backend boundary, by the backend it is addressed to. That keeps OpenCode's user-defined variant names usable from every entry point while a typo on the other backends still fails fast with the list of valid levels. The requested level is recorded in every episode's metadata (effort).Web workbench
reasoning_efforts), falling back to the backend's documented set.owner.jsonrole_configs.Practical tests
Real full MEA runs to completion (task executed, audited, verified) on three backends:
--effort low): all 10 spawned episodes carriedCLAUDE_CODE_EFFORT_LEVEL=lowin their commands and"effort": "low"in metadata; an unsupported level (--effort minimal) fails at agent setup:Claude Code effort must be one of low, medium, high, xhigh, max.@deepseek-ai/dsh0.1.0-rc.7, real DeepSeek API,--effort high): all 12 episodes carried--reasoning-effort high; every on-disk profile patch containsreasoningEffort: "high", anddsh --dump-configconfirms it merges into thellm-deepseekplugin config;--effort mediumfails at agent setup withDeepSeek Harness effort must be one of low, high, max.--effort highresolving to a custom varianthigh = { reasoningEffort: "high" }in the workspaceopencode.jsonc): all 10 episodes carried--variant highand"effort": "high"/"opencode_variant": "high"in metadata. A request-capturing endpoint confirmed the variant's options land in the API request body as"reasoning_effort". Note: OpenCode itself silently ignores a variant name the model does not define (no error) — upstream behavior, outside this harness.--effort deep, defined only in the workspaceopencode.jsonc): full MEA run from the CLI to completion — all 14 episodes carried--variant deepand"effort": "deep"in metadata, while--agent claude_code --effort deepstill fails at agent setup naming the valid levels.Docs & tests
effortrow in the[run]table, updated[run.roles.*]fallback chain, and an "Effort levels" subsection with the per-backend table.mainwith the frontend bundle built: 281 passed, 1 skipped; frontend core suite: 47 passed;tsc/vite build clean.