Skip to content

feat(computer-use): add guarded Windows maka.cu/2 integration - #4668

Open
sunheyi6 wants to merge 11 commits into
apache:mainfrom
sunheyi6:codex/windows-cu2-integration-v2
Open

feat(computer-use): add guarded Windows maka.cu/2 integration#4668
sunheyi6 wants to merge 11 commits into
apache:mainfrom
sunheyi6:codex/windows-cu2-integration-v2

Conversation

@sunheyi6

@sunheyi6 sunheyi6 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

This replacement is rebuilt from the current apache/main after #4497 and supersedes the oversized integration attempt in #4595.

  • Reuse the existing MakaCuService and shared maka.cu/2 backend on Windows; no second protocol or duplicate Windows service.
  • Select the strict background-only Windows executor, whose mutation surface is limited to semantic click and set_value. Keyboard, point, launch, scroll, text-selection, and foreground/global-input fallbacks are excluded.
  • Fail packaging closed: a non-ready helper must be absent, while a ready helper must be present and match the exact pinned file set, sizes, and SHA-256 digests.
  • Remove the local provenance-boolean escape hatch. Local preparation always writes distributionReady: false; only a separately reviewed release verifier may qualify an immutable artifact.
  • Require a valid Authenticode signature during packaged Windows verification when readiness is true.
  • Build from source with explicit x86_64-pc-windows-msvc, --locked, release, and static CRT contract metadata.

Why browser automation is excluded

Browser workflows remain owned by Maka Browser Use/OpenCLI. It can use browser-native DOM/accessibility, tab, navigation, page lifecycle, and command state, which are more reliable than desktop-window automation. Reimplementing browsers here would duplicate capability, widen permissions and test scope, and encourage coordinate/global-input fallbacks that conflict with the strict background-only invariant. This integration is for native desktop applications only.

Validation

  • node scripts/prepare-windows-cu-helper.test.mjs — 4 passed
  • New packaged Windows helper manifest/readiness test — passed
  • New product release fail-closed packaging test — passed
  • Biome check for all changed source/test files — passed
  • Companion native executor: 14 Rust tests, strict Clippy, explicit Windows release build, real handshake, and static CRT import inspection — passed

The broader product suites still expose three pre-existing environment/generated-output failures: Windows paths passed to WSL bash, missing @maka/eval/dist, and missing packages/runtime/dist/filesystem-worker/protocol.js. None originates in the changed Windows Computer Use files.

Release boundary

distributionReady remains false. This PR does not claim Authenticode, clean-machine, concurrent-user/mixed-DPI, or packaged real-conversation acceptance; those gates must qualify the same immutable artifact before release.

中文说明

摘要

本 replacement 基于包含 #4497 的当前 apache/main 重建,用于替代 #4595 中体量过大的集成尝试。

  • Windows 复用现有 MakaCuService 和共享 maka.cu/2 后端,不新增第二套协议或 service。
  • 接入严格后台执行器,修改能力只保留语义 clickset_value;排除键盘、点位、启动应用、滚动、文本选择以及任何前台/全局输入兜底。
  • 打包默认失败关闭:未就绪 helper 必须不存在;已就绪 helper 必须存在,并与固定文件集合、大小和 SHA-256 完全一致。
  • 删除本地 provenance 布尔逃生口。本地准备命令始终写入 distributionReady: false,只有独立审查的发布验证器才能认定不可变 artifact 合格。
  • readiness 为 true 时,Windows 打包验证强制要求有效 Authenticode 签名。
  • 源码构建明确使用 x86_64-pc-windows-msvc--locked、release 和静态 CRT 契约。

为什么排除浏览器

浏览器流程继续由 Maka Browser Use/OpenCLI 负责。它能使用浏览器原生 DOM/可访问性树、标签页、导航、页面生命周期和命令状态,比桌面窗口自动化更可靠。在这里重复实现浏览器会造成能力重复、扩大权限与测试范围,并诱导加入与严格后台约束冲突的坐标/全局输入兜底。因此本集成只服务原生桌面应用。

验证

  • helper 准备测试 4 项通过;新增打包 manifest/readiness 测试和产品发布失败关闭测试通过。
  • 所有变更源文件/测试的 Biome 检查通过。
  • 配套 native executor 的 14 项 Rust 测试、严格 Clippy、明确 Windows release 构建、真实握手和静态 CRT 导入检查均通过。

较大的产品测试仍暴露三个既有环境/生成物问题:Windows 路径传给 WSL bash、缺少 @maka/eval/dist、缺少 packages/runtime/dist/filesystem-worker/protocol.js;均不来自本次 Windows Computer Use 修改。

发布边界

distributionReady 仍为 false。本 PR 不宣称已完成 Authenticode、clean-machine、并发用户/混合 DPI 或 packaged real-conversation 验收;发布前必须针对同一份不可变 artifact 完成这些门禁。

@github-actions github-actions Bot added the effort/L Under 1000 readable lines label Sep 3, 2026
@sunheyi6

sunheyi6 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

@M4n5ter @hqhq1025 cc

@sunheyi6

sunheyi6 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Codex-assisted scope review, posted on behalf of @sunheyi6. The intended Windows product contract is now background-only desktop automation: it must not take over the user's active window, keyboard, mouse, or clipboard. Browser workflows are explicitly routed to Browser Use/OpenCLI because that layer can use browser-native page/DOM/accessibility state, tabs, navigation, and commands with stronger targeting and verification. Reimplementing browser control here would duplicate the browser stack and pressure the desktop executor to add coordinate, global-keyboard, or foreground fallbacks that violate the non-interference guarantee. This is an ownership boundary, not a loss of browser capability. The full design update is tracked in #3785.

This PR remains a useful guarded host/packaging scaffold while windowsCu.distributionReady is false. Before enabling distribution, the host/release side should enforce the following gates rather than relying on declared provenance alone:

  • when Windows distribution is enabled, a missing helper must fail packaging instead of being silently omitted;
  • verify the exact packaged helper file set, sizes, and SHA-256 digests;
  • perform real Authenticode verification for the exact artifact;
  • require a locked build with an explicit Windows target and either a genuinely self-contained binary or a verified set of shipped runtime dependencies;
  • require clean-machine packaged conversation E2E evidence for the background-only contract;
  • keep distributionReady: false until those checks are implemented and passing.

The E2E release gate must keep a user typing in a foreground app while Maka manipulates a different background desktop app, and verify that foreground HWND/PID, pointer position, clipboard, and user keystroke destination remain unchanged. A typed refusal is acceptable; silent foreground/global-input fallback is not.

No second host service, browser path, foreground compatibility mode, or duplicate public protocol is needed in this PR. The minimal integration is the existing shared service plus the qualified Windows helper.

中文说明

这是一条由 Codex 协助、代表 @sunheyi6 发布的范围审查说明。Windows 的产品合同现明确为只做后台桌面自动化:不能抢占用户当前窗口、键盘、鼠标或剪贴板。浏览器工作流明确路由给 Browser Use/OpenCLI,因为它可以直接使用页面/DOM/无障碍状态、标签页、导航和浏览器命令,定位与结果验证都更可靠。如果这里再实现浏览器控制,不但重复建设,还会迫使桌面执行器加入坐标点击、全局键盘或抢前台 fallback,破坏不干扰保证。这是职责边界,不是丢失浏览器能力。完整设计更新记录在 #3785

只要 windowsCu.distributionReady 仍为 false,本 PR 可以作为有保护的 Host/打包脚手架保留。在允许分发前,Host/发布侧必须实际执行以下门禁,不能只相信 provenance 中声明的字段:

  • 开启 Windows 分发时,helper 缺失必须导致打包失败,不能静默跳过;
  • 校验安装包内准确的 helper 文件集合、大小和 SHA-256;
  • 对确切产物执行真实 Authenticode 验证;
  • 使用锁定依赖和明确 Windows target 构建,并保证二进制确实自包含,或随包携带且验证全部运行时依赖;
  • 在干净机器上通过满足严格后台合同的真实安装包会话 E2E;
  • 在这些检查尚未实现并通过前,保持 distributionReady: false

发布 E2E 必须让用户持续在前台应用打字,同时 Maka 操作另一个后台桌面应用,并验证前台 HWND/PID、鼠标位置、剪贴板和用户按键去向都不变。明确拒绝是可接受结果;静默降级到前台/全局输入不可接受。

本 PR 不需要增加第二套 Host Service、浏览器路径、前台兼容模式或重复的公开协议。最小集成就是现有共享 Service 加上通过资格验证的 Windows helper。

sunheyi6 and others added 7 commits September 4, 2026 17:18
Use homogeneous fixed-length arrays for window position and size. Draft-07 tuple items arrays are rejected by JSON Schema 2020-12 consumers. Keep signed integer coordinates, positive sizes and strict two-item validation.

Add wire-schema and real Desktop descriptor coverage. No local artifact pins are included.

<details>
<summary>中文翻译</summary>

窗口位置和尺寸改为同类型定长数组,避免 draft-07 元组的 items 数组被 JSON Schema 2020-12 消费端拒绝。保留有符号整数坐标、正数尺寸和严格的两个元素校验。

补充 wire schema 和真实 Desktop 描述符测试,不包含本地制品绑定。

</details>
Reobserve unknown outcomes even when an older executor reports path:none. Consume snapshots when the executor reports numeric snapshotSpent:1, validate the field, and activate the final observation returned by element_sequence.

This accompanies the Windows executor policy change from requiring a background target to avoiding active focus/input interference. Users must be able to bring a window forward to watch its effects. Identity checks and no global keyboard/pointer fallback remain; unknown helper results are not promoted to verified.

Validation: 106 runtime/protocol tests and incremental compilation passed. A local full-permission Maka task created a SunCode conversation, entered and sent a greeting, and read the response with SunCode foreground and visible. Helper actions remained unknown; subsequent observations confirmed the application result. Windows backend mock tests were blocked by spawn EFTYPE. Automatic admission and packaged/clean-machine release qualification remain outside this evidence. No local binary hashes or release-readiness flags are changed.

<details>
<summary>中文翻译</summary>

即使旧执行器返回 path:none,未知结果也必须重新观察。根据执行器的数字 snapshotSpent:1 消耗快照并校验该字段,同时正确激活 element_sequence 返回的最终观察。

此修复配合 Windows 执行器从“目标必须后台”调整为“不主动干扰焦点与输入”:用户将窗口置前查看效果,不应导致操作被拒绝。身份校验及禁止全局键盘鼠标回退仍保留,不把 helper 的 unknown 提升为已验证成功。

验证:106 项 runtime/protocol 测试及增量编译通过。本地完全权限 Maka 任务在 SunCode 前台可见时完成新建对话、输入发送问候语和读取回复;helper 动作保持 unknown,由后续观察确认应用效果。Windows backend 模拟测试受 spawn EFTYPE 阻塞。自动权限准入、打包和干净机器发布验证不在本次成功范围内。不修改本机制品哈希或发布就绪标志。

</details>
Clarify that deferred tool activation is scoped to the current turn. Old activation history is not permission to issue unavailable calls or substitute Bash placeholders. Add failure-to-next-turn regression coverage.

This is model guidance, not a deterministic general loop breaker.

<details>
<summary>中文翻译</summary>

明确延迟工具激活仅在当前轮次有效。旧历史中的激活记录不能授权当前不存在的工具调用,也不能用 Bash 占位调用替代。补充失败后进入下一轮的回归测试。

这是模型提示改进,不是确定性的通用循环终止机制。

</details>
@sunheyi6

sunheyi6 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 40784e6 (schema compatibility), 754a41d (unknown outcomes and snapshot lifecycle), and 90aa98c (turn-scoped tool guidance). Companion executor fix: maka-agent/maka-cu#8, commit f8e9fde. The general Electron user-command environment fix is intentionally excluded from this PR and pushed separately to sunheyi6/maka-agent:codex/fix-user-command-electron-env; that branch does not yet have a new PR.

Why the strict-background policy changed

The previous Windows executor refused a target merely because the user brought its window to the foreground to watch the result. The product contract now prohibits actively disturbing focus and input, rather than requiring the target to remain in the background. Semantic UIA actions can run on an already-foreground window.

Process/window/element identity checks and desktop-state monitoring remain. There is no new activation, focus-stealing, physical-pointer movement, global keyboard injection or fallback path. Uncertain actions remain unknown; subsequent observation confirms application effects without promoting helper verification.

Validation and release scope

A local full-permission Maka conversation created a SunCode conversation, entered and sent the requested greeting, and read the response while SunCode was foreground and visible. The helper actions remained unknown, while fresh observations confirmed the completed workflow. Windows MSVC tests: 17 passed; runtime/protocol tests: 106 passed. Windows backend mock tests were blocked by spawn EFTYPE.

This is product source repair, not a test-only policy bypass. Local executables, profiles, logs and temporary artifact hashes were not pushed. Formal artifact pinning, automatic permission admission, packaged/clean-machine E2E and broader concurrent-user/application coverage remain unqualified; distributionReady remains false.

中文翻译

已推送 40784e6(参数兼容)、754a41d39(未知结果与观察生命周期)和 90aa98c(轮次工具提示)。配套原生执行器修复位于 maka-cu PR #8 的 f8e9fde。通用 Electron 用户命令环境修复刻意不放入本 PR,单独推送至 sunheyi6/maka-agent 的 codex/fix-user-command-electron-env 分支,目前尚未新建该分支的 PR。

为什么调整严格后台策略

原 Windows 执行器仅因用户将目标窗口置前查看操作效果,就拒绝操作。本次将产品约束从“目标必须保持后台”调整为“执行器不得主动干扰焦点与输入”,允许对已经处于前台的窗口执行 UIA 语义操作。

保留进程、窗口和元素身份校验及桌面状态监测,不新增窗口激活、抢焦点、物理鼠标移动、全局键盘注入或回退路径。不确定的动作仍返回 unknown,通过后续观察确认应用效果,不将 helper 验证升级为成功。

验证及发布范围

本地完全权限 Maka 对话在 SunCode 前台可见时完成新建对话、输入发送指定问候语和读取回复。helper 动作保持 unknown,由新的观察确认流程已完成。Windows MSVC 测试 17 项通过,runtime/protocol 测试 106 项通过;Windows backend 模拟测试受 spawn EFTYPE 阻塞。

这是产品源码修复,不是测试专用的策略绕过。本地执行器、测试配置、日志及临时制品哈希均未推送。正式制品绑定、自动权限准入、打包和干净机器 E2E,以及更广泛的并发用户和应用覆盖仍未完成验证;distributionReady 保持 false。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/L Under 1000 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant