- AI Desktop Agent — Universal Smart Pipeline
- Works with any AI provider · Runs free with local models · Self-healing doctor
+ OS-level desktop automation server. Gives any AI model eyes, hands, and ears on a real computer.
+ Model-agnostic · Works with Claude, GPT, Gemini, Llama, or any tool-calling model · Free with local models
@@ -14,83 +14,132 @@
- Website · Discord · Quick Start · How It Works · API · Changelog + Website · Discord · Quick Start · Connect · How It Works · API · Changelog
--- -## What's New in v0.6.3 +## What's New in v0.7.0 + +**Architecture overhaul. Universal tool server. True independence.** + +- **6-layer smart pipeline** — L0 (Browser) -> L1 (Action Router) -> L1.5 (Deterministic Flows) -> L2 (A11y Reasoner + CDP) -> L2.5 (Vision Hints) -> L3 (Computer Use). Most tasks never reach L3. +- **40 universal tools** — served via REST (`GET /tools`, `POST /execute/:name`) and MCP stdio from a single definition. Any model that can call functions can control your desktop. +- **3 transport modes** — `start` (full agent + tools), `serve` (tools only, bring your own brain), `mcp` (MCP stdio for Claude Code, Cursor, Windsurf, Zed) +- **CDP browser integration** — Chrome DevTools Protocol for DOM interaction, text extraction, click-by-selector. Auto-connects to Edge/Chrome. +- **Action verifier** — ground-truth checking after every action. Blocks false success reports. +- **A11y click resolver** — bounds-based coordinate resolution, zero LLM cost +- **Deterministic flows** — hardcoded keyboard sequences for common tasks (email compose, app switch). Zero LLM calls, instant. +- **No-progress loop detector** — blocks same action repeated 3+ times. Forces the LLM to try something different. +- **Premature-done blocker** — evidence-based completion checking. Won't report success unless verified. +- **Structured task logging** — JSONL per-task logs with `verified_success` vs `unverified_success` distinction +- **First-run onboarding** — consent flow explains what desktop control means before tools activate +- **Standalone data directory** — all data in `~/.clawdcursor/` (migrates from legacy paths automatically) +- **Error reporting** (opt-in) — `clawdcursor report` lets users send redacted task logs to help improve the agent + +### v0.6.3 vs v0.7.0 + +| | v0.6.3 | v0.7.0 | +|---|---|---| +| **Architecture** | 4-layer pipeline (L0-L3) | 6-layer pipeline (L0, L1, L1.5, L2, L2.5, L3) | +| **Transport** | REST API only | REST + MCP stdio + tools-only server | +| **Tools** | Monolithic agent, no tool exposure | 40 discrete tools, OpenAI function-calling format | +| **Browser** | Playwright-only, no DOM access | CDP integration — click by selector, read text, type by label | +| **Verification** | LLM self-reports success (often wrong) | Ground-truth action verifier — reads actual content back | +| **False positives** | Common — agent says "done" prematurely | Premature-done blocker + evidence-based completion | +| **Loops** | Agent can repeat same failed action forever | No-progress detector blocks after 3 repeats in 8 steps | +| **Click resolution** | Vision model guesses coordinates | A11y bounds-based resolver (zero LLM cost), vision as fallback | +| **Common tasks** | Every task goes through LLM | Deterministic flows for email, app-switch — zero LLM calls | +| **Task logging** | Console output only | Structured JSONL per-task, verified vs unverified success | +| **Data directory** | `~/.openclaw/clawdcursor/` (coupled) | `~/.clawdcursor/` (standalone, auto-migrates) | +| **Dependencies** | Tied to OpenClaw platform | Fully standalone — works with any AI, any client | +| **Onboarding** | None — starts immediately | First-run consent flow for desktop control | +| **MCP support** | None | Native MCP stdio for Claude Code, Cursor, Windsurf, Zed | +| **Error reporting** | None | Opt-in redacted task log submission | +| **Model coupling** | Anthropic-favored defaults | Truly model-agnostic — Claude, GPT, Gemini, Llama, Ollama, anything | -**Universal Pipeline, Multi-App Workflows, Provider-Agnostic.** +--- -- **🧠 LLM-based task pre-processor** — one cheap text LLM call decomposes any command into structured intent. No more brittle regex parsing. -- **📋 Multi-app workflows** — copy from Wikipedia, paste in Notepad? Works. 6-checkpoint tracking ensures every step completes (select → copy → switch app → click → paste → verify). -- **⌨️ Site-specific shortcuts** — Reddit (j/k/a/c), Twitter/X, YouTube, Gmail, GitHub, Slack + generic hints. Vision LLM uses keyboard instead of slow mouse clicks. -- **🌐 OS-level browser detection** — reads Windows registry or macOS LaunchServices for actual default browser. No hardcoded Edge/Safari. -- **🔄 3 smart verification retries** — on failure, builds step log digest + checkpoint status so the vision LLM fixes the exact missed step. -- **🔌 Mixed-provider pipelines** — kimi for text + anthropic for Computer Use, with per-layer API key resolution from OpenClaw auth-profiles. -- **🔧 Global install fix** — config discovery now checks package dir first, then cwd. -- **🏗️ Provider-agnostic internals** — no hardcoded model names, no hardcoded app lists, universal checkpoint detection. +## The Glove for Any AI Hand -## What's New in v0.6.1 +Think of your AI as the **hand** and Clawd Cursor as the **glove**. -**Keyboard Shortcuts, Pipeline Fixes, Better URL Handling.** +The hand has the intelligence — it reasons, plans, and decides what to do. The glove gives it grip on the physical world. Clawd Cursor wraps your entire desktop — every window, every button, every text field, every pixel — and exposes it as simple tool calls that any AI model can use. -- **⌨️ Keyboard shortcuts registry** — common actions (scroll, copy, reddit upvote) execute as direct keystrokes. Zero LLM calls, instant. -- **🔧 Pipeline gate fix** — Action Router now always runs, even for browser-context tasks. Shortcuts work everywhere. -- **🌐 Smarter URL extraction** — "open gmail and send email to foo@bar.com" correctly navigates to Gmail instead of bar.com. -- **🔄 CDP→UIDriver fallback** — Smart Interaction falls back to accessibility tree when browser CDP fails. -- **🛑 Reliable force-stop** — `clawdcursor stop` kills lingering processes. -- **📊 Provider label inference** — startup logs show text/vision providers clearly. +**Your AI is the brain. Clawd Cursor is the body.** -## What's New in v0.6.0 +If it's visible on your screen, Clawd Cursor can interact with it. Native apps, web apps, legacy software, internal tools, desktop games — anything with a GUI. No app-specific integrations needed. No APIs to configure per-service. One universal interface that turns any AI into a desktop operator. -**Universal Provider Support, OpenClaw Integration, Security Hardening.** +This is what makes v0.7.0 different from every other automation tool: **it doesn't care which AI drives it.** Claude, GPT, Gemini, Llama running locally, a custom model you trained yourself, or a simple Python script making function calls. If it can call tools, it can control your computer. -- **🔗 OpenClaw integration** — auto-discovers all configured providers from OpenClaw's config. No separate API key needed when running as a skill. -- **🌐 Universal provider support** — Anthropic, OpenAI, Groq, Together AI, DeepSeek, Kimi, Ollama, or any OpenAI-compatible endpoint. Provider auto-detected from API key format. -- **🧠 Mixed provider pipelines** — use Ollama for text (free) + cloud for vision (best quality). Doctor picks the optimal split automatically. -- **🔒 Security hardened** — sensitive app policy (agents must ask before email/banking/messaging), safety tiers enforced, no credentials stored in skill files. -- **🔧 Auto-detection as default** — no hardcoded models or providers. Doctor dynamically picks the best available setup. +``` +Your AI (any model) Clawd Cursor (the glove) + "Click the Send button" -> find_element + mouse_click + "What's on screen?" -> desktop_screenshot + read_screen + "Type my email" -> type_text + "Open Chrome to gmail" -> open_app + navigate_browser + "Read that table" -> cdp_read_text +``` -### v0.5.6 — Fluid Decomposition, Interactive Doctor, Smart Vision Fallback +--- -- **🧠 Fluid task decomposition** — LLM reasons about what ANY app needs instead of matching hardcoded patterns. -- **🩺 Interactive doctor** — scans all providers, detects GPU/VRAM, lets you pick TEXT and VISION LLMs. -- **🖥️ Smart vision fallback** — remaining subtasks bundled and handed to vision when cheap layers fail midway. +## Three Ways to Connect -### v0.5.2 — Web Dashboard + Browser Foreground Focus +Clawd Cursor is a **tool server**. It doesn't care which AI model drives it. -- **🖥️ Web Dashboard** — real-time logs, approve/reject safety confirmations, kill switch. Dark theme, zero dependencies. -- **🪟 Browser foreground focus** — Playwright activates Chrome at OS level. No more invisible background tabs. -- **Multi-provider** — 7+ providers supported out of the box -- **95% cheaper** — simple tasks run for $0 with local models -- **Self-healing** — if a model fails, the pipeline adapts automatically +### 1. Built-in Agent (`start`) -### Performance +Full autonomous agent with built-in LLM pipeline. Send a task, get a result. -| Task | v0.4 (single provider) | v0.5+ (local, $0) | v0.5+ (cloud) | -|------|-----------------------|---------------------|-------------------| -| Calculator (255*38=) | 43s | **2.6s** | **20.1s** | -| Notepad (type hello) | 73s | **2.0s** | **54.2s** | -| File Explorer | 53s | **1.9s** | **22.1s** | -| Gmail compose | 162s (18 LLM calls) | — | **21.7s** (1 LLM call) | +```bash +clawdcursor start +curl http://localhost:3847/task -H "Content-Type: application/json" \ + -d '{"task": "Open Notepad and write a haiku about the ocean"}' +``` ---- +### 2. Tools-Only Server (`serve`) -## OpenClaw Integration +Exposes 40 desktop tools via REST API. **You** bring the brain — Claude, GPT, Gemini, Llama, a script, anything. -Clawd Cursor ships as an [OpenClaw](https://openclaw.ai) skill. Install it and any OpenClaw agent — yours or community-built — can control your desktop through natural language. +```bash +clawdcursor serve -The [`SKILL.md`](SKILL.md) teaches agents **when and how** to use Clawd Cursor: REST API for full desktop control, CDP direct for fast browser reads. Agents learn to be independent — no more asking you to screenshot or copy-paste things they can do themselves. +# Discover available tools (OpenAI function-calling format) +curl http://localhost:3847/tools -For orchestration best practices (how to avoid overlap and keep OpenClaw + Clawd Cursor efficient), see [docs/OPENCLAW-INTEGRATION-RECOMMENDATIONS.md](docs/OPENCLAW-INTEGRATION-RECOMMENDATIONS.md). +# Execute any tool +curl http://localhost:3847/execute/desktop_screenshot +curl http://localhost:3847/execute/mouse_click -d '{"x": 500, "y": 300}' +curl http://localhost:3847/execute/type_text -d '{"text": "Hello world"}' +``` -```bash -# Install as OpenClaw skill -openclaw skills install clawd-cursor +### 3. MCP Mode (`mcp`) + +Runs as an MCP tool server over stdio. Works with Claude Code, Cursor, Windsurf, Zed, or any MCP-compatible client. + +```jsonc +// Claude Code: ~/.claude/settings.json +{ + "mcpServers": { + "clawdcursor": { + "command": "node", + "args": ["/path/to/clawdcursor/dist/index.js", "mcp"] + } + } +} ``` +### Tool Categories (40 tools) + +| Category | Tools | Examples | +|----------|-------|---------| +| Perception | 9 | `desktop_screenshot`, `read_screen`, `get_active_window`, `get_focused_element`, `smart_read`, `ocr_read_screen` | +| Mouse | 6 | `mouse_click`, `mouse_double_click`, `mouse_drag`, `mouse_scroll` | +| Keyboard | 5 | `key_press`, `type_text`, `smart_type`, `shortcuts_list`, `shortcuts_execute` | +| Window/App | 6 | `focus_window`, `open_app`, `get_windows`, `invoke_element` | +| Browser CDP | 10 | `cdp_connect`, `cdp_click`, `cdp_type`, `cdp_read_text` | +| Orchestration | 4 | `delegate_to_agent`, `smart_click`, `navigate_browser`, `wait` | + --- ## Quick Start @@ -98,278 +147,143 @@ openclaw skills install clawd-cursor ### Windows ```powershell -git clone https://github.com/AmrDab/clawd-cursor.git -cd clawd-cursor +git clone https://github.com/AmrDab/clawdcursor.git +cd clawdcursor npm install npm run setup # builds + registers 'clawdcursor' command globally -# Just install and start — auto-configures from OpenClaw or env vars +# Start the full agent clawdcursor start -# Or specify any provider -clawdcursor start --base-url https://api.example.com/v1 --api-key KEY +# Or start tools-only (bring your own AI) +clawdcursor serve -# Fine-tune setup interactively (optional) -clawdcursor doctor +# Or run as MCP server +clawdcursor mcp ``` ### macOS ```bash -git clone https://github.com/AmrDab/clawd-cursor.git -cd clawd-cursor && npm install && npm run setup +git clone https://github.com/AmrDab/clawdcursor.git +cd clawdcursor && npm install && npm run setup # Grant Accessibility permissions to your terminal first! -# System Settings → Privacy & Security → Accessibility → Add Terminal/iTerm +# System Settings -> Privacy & Security -> Accessibility -> Add Terminal/iTerm -# Make macOS scripts executable chmod +x scripts/mac/*.sh scripts/mac/*.jxa - -# Just start — auto-detects available providers clawdcursor start - -# Or specify any provider -clawdcursor start --base-url https://api.example.com/v1 --api-key KEY ``` ### Linux ```bash -git clone https://github.com/AmrDab/clawd-cursor.git -cd clawd-cursor && npm install && npm run setup +git clone https://github.com/AmrDab/clawdcursor.git +cd clawdcursor && npm install && npm run setup -# Linux: browser control via CDP only (no native desktop automation) -# Just start — auto-detects available providers +# Linux: browser control via CDP only (no native desktop automation yet) clawdcursor start - -# Or specify any provider -clawdcursor start --base-url https://api.example.com/v1 --api-key KEY ``` -> 📖 See [docs/MACOS-SETUP.md](docs/MACOS-SETUP.md) for the full macOS onboarding guide. - -First run auto-configuration will: -1. Scan for AI providers from OpenClaw config, environment variables, and CLI flags -2. Quick-test discovered providers (5s timeout per provider) -3. Build the optimal pipeline automatically -4. Save config and start immediately - -The optional `doctor` command provides interactive configuration: -1. Tests your screen capture and accessibility bridge -2. Scans all AI providers (Anthropic, OpenAI, Groq, Together, DeepSeek, Kimi, Ollama) and detects GPU/VRAM -3. Tests each model and shows you what works with latency -4. Lets you pick your TEXT LLM and VISION LLM (or accept the recommended defaults) -5. Shows setup instructions for any unconfigured cloud providers -6. Builds your optimal pipeline and saves it - -Send a task: -```bash -clawdcursor task "Open Notepad and type hello world" - -# Or via API: -curl http://localhost:3847/task -H "Content-Type: application/json" \ - -d '{"task": "Open Notepad and type hello world"}' -``` +> See [docs/MACOS-SETUP.md](docs/MACOS-SETUP.md) for the full macOS onboarding guide. -> **Note:** `npm run setup` runs `npm run build && npm link`, which registers `clawdcursor` as a global command. If you prefer not to link globally, run `npm run build` instead and use `npx clawdcursor` or `node dist/index.js` to run commands. +First run will: +1. Show a desktop control consent warning (one-time) +2. Scan for AI providers from environment variables and CLI flags +3. Auto-configure the optimal pipeline +4. Start the server on `http://localhost:3847` -### Provider Quick Setup +### Provider Setup **Free (no API key needed):** ```bash -# Just need Ollama running with any model -ollama pullGive your agent a screen. Gmail, Slack, Jira, Figma - if you can click it, your agent can too. Native desktop control. No servers. No API keys per service.
- -Every app your agent needs already has a UI. One skill gives your agent eyes and hands on all of them.
-Each service needs its own API key, auth flow, rate limit handling, and breaking change maintenance.
-Your agent sees the screen and interacts with any app the same way you do. Zero per-service setup.
-No OAuth flows, no token rotation, no webhook endpoints. If you are logged in, your agent is too.
-Three pipeline layers. Each task falls to the cheapest layer that can handle it.
-No cherry-picking. v0.5.1 smart pipeline benchmarks.
-Queries native accessibility APIs to find buttons, menus, inputs by name - not pixel coordinates.
-Common tasks execute in milliseconds with zero AI calls via UI Automation.
-Claude gets native desktop tools - screenshot, click, type, scroll, drag. Fully autonomous.
-Scaled to 1280×720 for the API. ~57ms native capture, HD quality for reliable icon identification. Adaptive delays per action type.
-Auto, Preview, Confirm. Three tiers from safe reads to destructive actions.
-Direct OS-level screen capture and input via @nut-tree-fork/nut-js. No server needed. 17× faster screenshots.
-Auto-detects your AI provider, tests models, configures the optimal pipeline. Falls back gracefully if a model is unavailable.
-Open apps, navigate, read screen. Executes immediately.
-Type text, fill forms. Logged before executing.
-Send messages, delete files. Waits for approval.
-Just point your AI assistant to clawdcursor.com - or run it manually:
- -git clone https://github.com/AmrDab/clawd-cursor.git
-cd clawd-cursor
-npm install; npm run setup
-clawdcursor doctorRequires: Node.js 20+. npm run setup builds and registers clawdcursor globally. Use npm run build if you prefer npx clawdcursor instead.
Open source. One install. Your agent sees your screen and gets to work.
- - - Star on GitHub - -Every release, from day one.
-Connect any AI to your desktop. It sees your screen, moves the mouse, types, clicks — anything you can do, it can do. Works with Claude, GPT, Gemini, Llama, or any AI.
+ +If it's on your screen, your AI can use it. Gmail, Slack, Figma, Jira, native apps, legacy software — anything with a UI.
+Bring an API key and just describe what you want. clawdcursor figures out the steps, executes them, and verifies they worked. You stay in plain English.
+Already using Claude Code, Cursor, or Windsurf? Add clawdcursor and your AI gets full desktop control as a native tool — no extra setup, no extra API calls.
+One server. Three modes. They all give your AI the same desktop access.
+clawdcursor consent onceclawdcursor doctor to set up your AIclawdcursor startclawdcursor startGET /toolsEach task falls to the cheapest layer that can handle it. OCR reads the screen as text. Vision is the absolute last resort. Only active in CLI agent mode.
+Pattern-matched tasks: open apps, type text, press shortcuts, navigate URLs. Zero LLM calls. Instant execution.
+CDPDriver + UIDriver combo. One cheap text LLM call plans a multi-step action sequence. No screenshots needed.
+Remembers known action patterns (ctrl+t = new tab). Skips perception entirely for previously-solved interactions.
+OCR reads screen text + bounding boxes. A11y tree supplements in parallel. Text model reasons and clicks by coordinate. Zero screenshots.
+Handles browser-specific tasks first via Playwright automation before escalating to the full pipeline.
+Full vision model with screenshot + action loop. Last resort for anything the text layers can't handle. Supports any visible UI.
+v0.7.0 makes clawdcursor usable by low-cost agents that can't afford screenshots. OCR reads the screen as text — vision is the last resort.
+Windows.Media.Ocr reads every pixel on screen as text with bounding boxes. A11y tree runs in parallel as a supplement. Zero screenshots for most tasks.
+4 new MCP tools: smart_read, smart_click, smart_type, invoke_element. Click buttons by name, not coordinates. No vision needed.
Built-in keyboard shortcut database. shortcuts_list discovers available shortcuts. shortcuts_execute fires them instantly.
Reads actual screen state after every action. Blocks false "done" reports — the agent can't lie about success anymore.
+Plug directly into Claude Code, Cursor, Windsurf, or Zed as an MCP server. No REST plumbing — tools show up natively in your AI's registry.
+Learns action patterns over time. Once it knows "new tab = ctrl+t", it skips all perception layers and executes instantly. Zero cost.
+Install, consent once, connect your AI.
+ +# 1. Install
+git clone https://github.com/AmrDab/clawdcursor.git --branch v0.7.0
+cd clawdcursor && npm install && npm run setup
+
+# 2. One-time consent (required — desktop control warning)
+clawdcursor consent
+
+# 3. Add to your MCP client config
+{
+ "mcpServers": {
+ "clawdcursor": {
+ "command": "clawdcursor",
+ "args": ["mcp"]
+ }
+ }
+}
+
+# clawdcursor mcp will refuse to start without consentRequires Node.js 20+. Consent is one-time and stored in ~/.clawdcursor/consent. Server binds to localhost only.
Open source. Any model. Any client. Your desktop, controlled.
+ + + Star on GitHub + +