feat(computer-use): AX UI locate and mouse precision tools#278
Merged
GCWing merged 4 commits intoGCWing:mainfrom Mar 26, 2026
Merged
feat(computer-use): AX UI locate and mouse precision tools#278GCWing merged 4 commits intoGCWing:mainfrom
GCWing merged 4 commits intoGCWing:mainfrom
Conversation
added 4 commits
March 26, 2026 18:56
- Platform AX/UI modules (macOS, Windows, Linux) and shared locate helpers - New tools: locate, mouse click/precise/step; registry and claw mode updates - Desktop host, config API, execution engine, bash tool integration - Web UI session config and i18n for computer use options
… capture - linux_ax_ui: import atspi::connection::P2P for object_as_accessible - desktop_host: gate macOS/Windows-only types and DisplayInfo; macOS-only chord_has_modifier - webdriver capture: move base64 imports into macOS/Windows imp (fix unused on Linux)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds platform AX/UI-based element locating (macOS, Windows, Linux), new computer-use tools (locate, mouse click/precise/step), and integrates them with claw mode, registry, desktop host, and session config.
Changes
linux_ax_ui,macos_ax_ui,windows_ax_ui,ui_locate_common; host and Cargo wiringcomputer_use_host, execution engine, bash tool, claw promptVerification
cargo check -p bitfun-desktoppassed locally.Branch is based on
upstream/mainwith a single commit (avoids unrelated local branch history).