Add browser-use CDP extraction as copyable tool

[wu-changxing]

Context

Our browser element extraction uses JS injection (extract_elements.js) + injected data-browser-agent-id attributes. This is simple and reliable but misses some cases that browser-use handles via Chrome DevTools Protocol (CDP).

What browser-use does better

• Event listener detection — catches React/Vue dynamically attached handlers
• Paint order occlusion — filters elements hidden behind other elements
• Tree structure — gives LLM parent/child context
• New element tracking — marks elements that appeared since last step

What our approach does better

• Simpler (one JS file + one Python file)
• Reliable CSS locators (data-browser-agent-id)
• Cross-browser (works with Firefox/WebKit via Playwright)

Proposal

Add browser-use's dom/ module as a co copy-able tool:

co copy browser_use_extractor

Users who need advanced extraction (complex SPAs, heavy shadow DOM usage) can swap it in. Keep our current approach as default.

Already Fixed

• Shadow DOM traversal (v0.8.8) — covers LinkedIn modals
• contenteditable detection (v0.8.8)
• Post-click wait for DOM changes (v0.8.8)

Reference

• browser-use/browser-use — CDP-based extraction
• Their key files: browser_use/dom/, browser_use/dom/buildDomTree.js

[wu-changxing]

🔍 Summary

Add browser-use's CDP-based DOM extraction as an opt-in copyable tool via co copy browser_use_extractor. This gives users who work with complex SPAs (React event listeners, modal overlays, nested iframes) a drop-in replacement for our default JS injection approach. Our current extract_elements.js stays as the default — simple, cross-browser, reliable.

📊 Current vs Proposed

CURRENT:
┌──────────┐   page.evaluate()   ┌──────────────────────┐
│  Agent   │ ──────────────────▶ │ extract_elements.js  │
└──────────┘                     │ (JS injection)       │
                                 │ ✅ Simple (235 lines) │
                                 │ ✅ Cross-browser      │
                                 │ ❌ Misses addEventListener handlers │
                                 │ ❌ No occlusion detection           │
                                 │ ❌ Flat list (no tree)              │
                                 └──────────────────────┘

PROPOSED (additive — current stays as default):
┌──────────┐   co copy browser_use_extractor
│  User    │ ──────────────────────────────▶ ./tools/browser_use_extractor/
└──────────┘                                 │
                                             ├── __init__.py
                                             ├── cdp_extractor.py      ← CDP tree builder
                                             ├── buildDomTree.js       ← browser-use JS (adapted)
                                             └── README.md             ← usage guide
                                             │
                                             │ User swaps ONE import:
                                             │   from tools.browser_use_extractor import extract_elements
                                             ▼
                                 ┌──────────────────────────┐
                                 │ CDP Protocol Extraction   │
                                 │ ✅ getEventListeners()    │
                                 │ ✅ Paint order occlusion  │
                                 │ ✅ Accessibility tree     │
                                 │ ✅ Tree structure for LLM │
                                 │ ⚠️  Chrome-only (CDP)     │
                                 │ ⚠️  More complex (~400 LOC)│
                                 └──────────────────────────┘

📁 Files to Change

connectonion/
├── connectonion/
│   ├── useful_tools/
│   │   └── browser_use_extractor/          ← NEW directory (copyable tool)
│   │       ├── __init__.py                 ← exports extract_elements(), same interface
│   │       ├── cdp_extractor.py            ← CDP-based extraction logic
│   │       ├── buildDomTree.js             ← adapted from browser-use (tree builder)
│   │       └── README.md                   ← usage instructions + swap guide
│   │
│   └── cli/
│       └── commands/
│           └── copy_commands.py            ← MODIFY: add to TOOLS registry
│
├── tests/
│   └── unit/
│       └── test_cdp_extractor.py           ← NEW: unit tests (mocked CDP)
│
└── setup.py                                ← MODIFY: include new package_data

🏗️ Architecture — Drop-in Interface

                    extract_elements(page) → List[InteractiveElement]
                              ▲
                              │ same interface
                    ┌─────────┴──────────┐
                    │                    │
         ┌─────────────────┐  ┌──────────────────────┐
         │  element_finder  │  │ browser_use_extractor │
         │  (default)       │  │ (opt-in via co copy)  │
         │                  │  │                        │
         │ JS injection     │  │ CDP protocol calls:    │
         │ page.evaluate()  │  │ 1. DOM.getDocument     │
         │                  │  │ 2. DOMSnapshot         │
         │ Returns:         │  │ 3. Accessibility tree  │
         │ InteractiveElement│ │ 4. getEventListeners   │
         │ (flat list)      │  │                        │
         └─────────────────┘  │ Returns:               │
                               │ InteractiveElement     │
                               │ (enriched, same type)  │
                               └──────────────────────┘

Key: Both return List[InteractiveElement] — drop-in replacement.
The user swaps ONE import in their browser agent code.

🔄 Data Flow — CDP Extraction Pipeline

extract_elements(page)
        │
        ▼
page.context.new_cdp_session(page)     ← Playwright CDP session
        │
        ├──────────────────────────────── Parallel CDP calls:
        │                                 │
        │  ┌─────────────────────┐  ┌─────────────────────┐
        │  │ DOM.getDocument     │  │ DOMSnapshot         │
        │  │ (full tree, depth=-1│  │ .captureSnapshot    │
        │  │  pierce=True)       │  │ (bounds, paint order│
        │  └────────┬────────────┘  │  computed styles)   │
        │           │               └────────┬────────────┘
        │           ▼                        ▼
        │  ┌─────────────────────┐  ┌─────────────────────┐
        │  │ Runtime.evaluate    │  │ Build lookup tables  │
        │  │ getEventListeners() │  │ backendNodeId → data │
        │  │ (DevTools-only API) │  └────────┬────────────┘
        │  └────────┬────────────┘           │
        │           │                        │
        │           ▼                        ▼
        │  ┌────────────────────────────────────┐
        │  │ Walk DOM tree + merge data sources │
        │  │ • Attach bounds from snapshot       │
        │  │ • Mark has_event_listener from CDP  │
        │  │ • Filter by visibility + occlusion  │
        │  │ • Inject data-browser-agent-id      │  ← keep our locator strategy!
        │  └───────────────┬────────────────────┘
        │                  │
        │                  ▼
        │  ┌────────────────────────────────────┐
        │  │ Convert to List[InteractiveElement] │
        │  │ Same shape as JS extraction output  │
        │  └────────────────────────────────────┘
        │
        ▼
  Return to caller (element_finder.find_element, browser.click, etc.)

📝 Implementation Steps

1. Create browser_use_extractor/ directory

   • connectonion/useful_tools/browser_use_extractor/
   • __init__.py exports extract_elements(page) → List[InteractiveElement]
   • Reuse InteractiveElement from existing element_finder.py (import it)

2. Write cdp_extractor.py — core CDP logic (~300-400 lines)

   • Open CDP session via page.context.new_cdp_session(page) (Playwright built-in, no extra deps)
   • Call DOM.getDocument(depth=-1, pierce=True) for full DOM tree
   • Call DOMSnapshot.captureSnapshot(includePaintOrder=True) for bounds + occlusion
   • Call Runtime.evaluate with includeCommandLineAPI=True for getEventListeners()
   • Walk tree, merge data, filter visible/non-occluded elements
   • Keep data-browser-agent-id injection — inject via DOM.setAttributeValue CDP call
   • Return List[InteractiveElement] matching existing interface
   • Skip for MVP: accessibility tree merge, cross-origin iframe piercing

3. Adapt buildDomTree.js from browser-use

   • Simplified version focused on tree structure + event listeners
   • Credit browser-use in header comment (MIT license)
   • Remove their custom node serialization — we convert to our InteractiveElement format

4. Add README.md with usage guide

   • How to swap in (one import change)
   • Chrome-only limitation documented
   • When to use this vs default (complex SPAs, React/Vue event handlers, modal-heavy UIs)

5. Register in co copy system

   • copy_commands.py: Add "browser_use_extractor": "browser_use_extractor" to TOOLS dict
   • Handle as directory copy (like PROMPTS pattern) since it's a multi-file tool

6. Update setup.py package_data

   • Include useful_tools/browser_use_extractor/*.js and useful_tools/browser_use_extractor/*.md

7. Write unit tests (tests/unit/test_cdp_extractor.py)

   • Mock CDP session responses
   • Test DOM tree walking + merge
   • Test occlusion filtering (use elementFromPoint() approach for MVP — ~5 lines vs browser-use's ~200 line geometry engine)
   • Test InteractiveElement output matches expected format

⚠️ Risks & Considerations

• Chrome-only: CDP is Chrome/Chromium only. Our default JS approach works with Firefox/WebKit too. Must document clearly.
• Playwright CDP API: page.context.new_cdp_session() is built-in — no extra dependency. But only works with Chromium contexts.
• License: browser-use is MIT licensed. We can adapt their JS with attribution.
• Scope creep: browser-use is ~2000+ lines across 8 files. We adapt the key wins (event listeners, basic occlusion) — target ~400 lines total.
• Paint order occlusion MVP: Use elementFromPoint() at element center instead of browser-use's full rect-union geometry engine. ~5 lines vs ~200 lines, covers 90% of cases.

🧪 Testing Strategy

• Unit tests: Mock CDP session, verify tree walking, occlusion filtering, output format
• Manual test: co copy browser_use_extractor → verify files appear in ./tools/
• Integration test: Side-by-side comparison on a React SPA to verify event listener detection
• Regression: Ensure element_finder.py still works unchanged (zero breaking changes)