feat(computer-use): add guarded Windows maka.cu/2 integration - #4668
feat(computer-use): add guarded Windows maka.cu/2 integration#4668sunheyi6 wants to merge 11 commits into
Conversation
|
Codex-assisted scope review, posted on behalf of @sunheyi6. The intended Windows product contract is now background-only desktop automation: it must not take over the user's active window, keyboard, mouse, or clipboard. Browser workflows are explicitly routed to Browser Use/OpenCLI because that layer can use browser-native page/DOM/accessibility state, tabs, navigation, and commands with stronger targeting and verification. Reimplementing browser control here would duplicate the browser stack and pressure the desktop executor to add coordinate, global-keyboard, or foreground fallbacks that violate the non-interference guarantee. This is an ownership boundary, not a loss of browser capability. The full design update is tracked in #3785. This PR remains a useful guarded host/packaging scaffold while
The E2E release gate must keep a user typing in a foreground app while Maka manipulates a different background desktop app, and verify that foreground HWND/PID, pointer position, clipboard, and user keystroke destination remain unchanged. A typed refusal is acceptable; silent foreground/global-input fallback is not. No second host service, browser path, foreground compatibility mode, or duplicate public protocol is needed in this PR. The minimal integration is the existing shared service plus the qualified Windows helper. 中文说明这是一条由 Codex 协助、代表 @sunheyi6 发布的范围审查说明。Windows 的产品合同现明确为只做后台桌面自动化:不能抢占用户当前窗口、键盘、鼠标或剪贴板。浏览器工作流明确路由给 Browser Use/OpenCLI,因为它可以直接使用页面/DOM/无障碍状态、标签页、导航和浏览器命令,定位与结果验证都更可靠。如果这里再实现浏览器控制,不但重复建设,还会迫使桌面执行器加入坐标点击、全局键盘或抢前台 fallback,破坏不干扰保证。这是职责边界,不是丢失浏览器能力。完整设计更新记录在 #3785。 只要
发布 E2E 必须让用户持续在前台应用打字,同时 Maka 操作另一个后台桌面应用,并验证前台 HWND/PID、鼠标位置、剪贴板和用户按键去向都不变。明确拒绝是可接受结果;静默降级到前台/全局输入不可接受。 本 PR 不需要增加第二套 Host Service、浏览器路径、前台兼容模式或重复的公开协议。最小集成就是现有共享 Service 加上通过资格验证的 Windows helper。 |
…ckground-release-gates
Use homogeneous fixed-length arrays for window position and size. Draft-07 tuple items arrays are rejected by JSON Schema 2020-12 consumers. Keep signed integer coordinates, positive sizes and strict two-item validation. Add wire-schema and real Desktop descriptor coverage. No local artifact pins are included. <details> <summary>中文翻译</summary> 窗口位置和尺寸改为同类型定长数组,避免 draft-07 元组的 items 数组被 JSON Schema 2020-12 消费端拒绝。保留有符号整数坐标、正数尺寸和严格的两个元素校验。 补充 wire schema 和真实 Desktop 描述符测试,不包含本地制品绑定。 </details>
Reobserve unknown outcomes even when an older executor reports path:none. Consume snapshots when the executor reports numeric snapshotSpent:1, validate the field, and activate the final observation returned by element_sequence. This accompanies the Windows executor policy change from requiring a background target to avoiding active focus/input interference. Users must be able to bring a window forward to watch its effects. Identity checks and no global keyboard/pointer fallback remain; unknown helper results are not promoted to verified. Validation: 106 runtime/protocol tests and incremental compilation passed. A local full-permission Maka task created a SunCode conversation, entered and sent a greeting, and read the response with SunCode foreground and visible. Helper actions remained unknown; subsequent observations confirmed the application result. Windows backend mock tests were blocked by spawn EFTYPE. Automatic admission and packaged/clean-machine release qualification remain outside this evidence. No local binary hashes or release-readiness flags are changed. <details> <summary>中文翻译</summary> 即使旧执行器返回 path:none,未知结果也必须重新观察。根据执行器的数字 snapshotSpent:1 消耗快照并校验该字段,同时正确激活 element_sequence 返回的最终观察。 此修复配合 Windows 执行器从“目标必须后台”调整为“不主动干扰焦点与输入”:用户将窗口置前查看效果,不应导致操作被拒绝。身份校验及禁止全局键盘鼠标回退仍保留,不把 helper 的 unknown 提升为已验证成功。 验证:106 项 runtime/protocol 测试及增量编译通过。本地完全权限 Maka 任务在 SunCode 前台可见时完成新建对话、输入发送问候语和读取回复;helper 动作保持 unknown,由后续观察确认应用效果。Windows backend 模拟测试受 spawn EFTYPE 阻塞。自动权限准入、打包和干净机器发布验证不在本次成功范围内。不修改本机制品哈希或发布就绪标志。 </details>
Clarify that deferred tool activation is scoped to the current turn. Old activation history is not permission to issue unavailable calls or substitute Bash placeholders. Add failure-to-next-turn regression coverage. This is model guidance, not a deterministic general loop breaker. <details> <summary>中文翻译</summary> 明确延迟工具激活仅在当前轮次有效。旧历史中的激活记录不能授权当前不存在的工具调用,也不能用 Bash 占位调用替代。补充失败后进入下一轮的回归测试。 这是模型提示改进,不是确定性的通用循环终止机制。 </details>
|
Pushed 40784e6 (schema compatibility), 754a41d (unknown outcomes and snapshot lifecycle), and 90aa98c (turn-scoped tool guidance). Companion executor fix: maka-agent/maka-cu#8, commit f8e9fde. The general Electron user-command environment fix is intentionally excluded from this PR and pushed separately to sunheyi6/maka-agent:codex/fix-user-command-electron-env; that branch does not yet have a new PR. Why the strict-background policy changedThe previous Windows executor refused a target merely because the user brought its window to the foreground to watch the result. The product contract now prohibits actively disturbing focus and input, rather than requiring the target to remain in the background. Semantic UIA actions can run on an already-foreground window. Process/window/element identity checks and desktop-state monitoring remain. There is no new activation, focus-stealing, physical-pointer movement, global keyboard injection or fallback path. Uncertain actions remain Validation and release scopeA local full-permission Maka conversation created a SunCode conversation, entered and sent the requested greeting, and read the response while SunCode was foreground and visible. The helper actions remained This is product source repair, not a test-only policy bypass. Local executables, profiles, logs and temporary artifact hashes were not pushed. Formal artifact pinning, automatic permission admission, packaged/clean-machine E2E and broader concurrent-user/application coverage remain unqualified; 中文翻译已推送 40784e6(参数兼容)、754a41d39(未知结果与观察生命周期)和 90aa98c(轮次工具提示)。配套原生执行器修复位于 maka-cu PR #8 的 f8e9fde。通用 Electron 用户命令环境修复刻意不放入本 PR,单独推送至 sunheyi6/maka-agent 的 codex/fix-user-command-electron-env 分支,目前尚未新建该分支的 PR。 为什么调整严格后台策略原 Windows 执行器仅因用户将目标窗口置前查看操作效果,就拒绝操作。本次将产品约束从“目标必须保持后台”调整为“执行器不得主动干扰焦点与输入”,允许对已经处于前台的窗口执行 UIA 语义操作。 保留进程、窗口和元素身份校验及桌面状态监测,不新增窗口激活、抢焦点、物理鼠标移动、全局键盘注入或回退路径。不确定的动作仍返回 unknown,通过后续观察确认应用效果,不将 helper 验证升级为成功。 验证及发布范围本地完全权限 Maka 对话在 SunCode 前台可见时完成新建对话、输入发送指定问候语和读取回复。helper 动作保持 unknown,由新的观察确认流程已完成。Windows MSVC 测试 17 项通过,runtime/protocol 测试 106 项通过;Windows backend 模拟测试受 spawn EFTYPE 阻塞。 这是产品源码修复,不是测试专用的策略绕过。本地执行器、测试配置、日志及临时制品哈希均未推送。正式制品绑定、自动权限准入、打包和干净机器 E2E,以及更广泛的并发用户和应用覆盖仍未完成验证;distributionReady 保持 false。 |
Summary
This replacement is rebuilt from the current
apache/mainafter #4497 and supersedes the oversized integration attempt in #4595.MakaCuServiceand sharedmaka.cu/2backend on Windows; no second protocol or duplicate Windows service.clickandset_value. Keyboard, point, launch, scroll, text-selection, and foreground/global-input fallbacks are excluded.distributionReady: false; only a separately reviewed release verifier may qualify an immutable artifact.x86_64-pc-windows-msvc,--locked, release, and static CRT contract metadata.Why browser automation is excluded
Browser workflows remain owned by Maka Browser Use/OpenCLI. It can use browser-native DOM/accessibility, tab, navigation, page lifecycle, and command state, which are more reliable than desktop-window automation. Reimplementing browsers here would duplicate capability, widen permissions and test scope, and encourage coordinate/global-input fallbacks that conflict with the strict background-only invariant. This integration is for native desktop applications only.
Validation
node scripts/prepare-windows-cu-helper.test.mjs— 4 passedThe broader product suites still expose three pre-existing environment/generated-output failures: Windows paths passed to WSL
bash, missing@maka/eval/dist, and missingpackages/runtime/dist/filesystem-worker/protocol.js. None originates in the changed Windows Computer Use files.Release boundary
distributionReadyremains false. This PR does not claim Authenticode, clean-machine, concurrent-user/mixed-DPI, or packaged real-conversation acceptance; those gates must qualify the same immutable artifact before release.中文说明
摘要
本 replacement 基于包含 #4497 的当前
apache/main重建,用于替代 #4595 中体量过大的集成尝试。MakaCuService和共享maka.cu/2后端,不新增第二套协议或 service。click和set_value;排除键盘、点位、启动应用、滚动、文本选择以及任何前台/全局输入兜底。distributionReady: false,只有独立审查的发布验证器才能认定不可变 artifact 合格。x86_64-pc-windows-msvc、--locked、release 和静态 CRT 契约。为什么排除浏览器
浏览器流程继续由 Maka Browser Use/OpenCLI 负责。它能使用浏览器原生 DOM/可访问性树、标签页、导航、页面生命周期和命令状态,比桌面窗口自动化更可靠。在这里重复实现浏览器会造成能力重复、扩大权限与测试范围,并诱导加入与严格后台约束冲突的坐标/全局输入兜底。因此本集成只服务原生桌面应用。
验证
较大的产品测试仍暴露三个既有环境/生成物问题:Windows 路径传给 WSL
bash、缺少@maka/eval/dist、缺少packages/runtime/dist/filesystem-worker/protocol.js;均不来自本次 Windows Computer Use 修改。发布边界
distributionReady仍为 false。本 PR 不宣称已完成 Authenticode、clean-machine、并发用户/混合 DPI 或 packaged real-conversation 验收;发布前必须针对同一份不可变 artifact 完成这些门禁。