feat: low-level interaction endpoints (mouse-wheel, init-script, capture-network, capture-requests)

[nayrosk]

Summary

Four new endpoints that fill gaps in the existing interaction surface, all needed in practice to drive modern anti-automation web apps (e.g. Instagram DMs):

1. POST /tabs/:tabId/mouse-wheel — real page.mouse.wheel() dispatched at a specific element or coordinate
2. POST /tabs/:tabId/init-script — wraps page.addInitScript() so a hook runs on every navigation before any page script
3. POST /tabs/:tabId/capture-network — wraps page.on("response") for a bounded duration, captures matching response bodies at the browser network layer (above the Service Worker, above any in-page closure)
4. POST /tabs/:tabId/capture-requests — wraps page.on("request") for the symmetric request side, exposing URL + method + POST body + headers (essential when the page bundle hides outgoing payloads behind closure-cached primitives)

Each addresses a distinct interception level. Together they cover the full stack: DOM events → page lifecycle → network (both directions).

---

1. /tabs/:tabId/mouse-wheel

POST /tabs/:tabId/mouse-wheel
{
  "userId": "...",
  "ref": "e22",          // optional: element ref → wheel at bbox centre
  "x": 810, "y": 363,    // optional: explicit page coords (ignored if ref is set)
  "deltaX": 0,
  "deltaY": -1500
}
→ { "ok": true, "x": 810, "y": 363, "deltaX": 0, "deltaY": -1500 }

The existing /scroll endpoint calls mouse.wheel() without prior cursor positioning, so it targets wherever the cursor happens to be — too coarse for nested scrollable containers. Some sites (notably Instagram DMs) virtualise their message lists and ignore both programmatic scrollTop and JS-dispatched WheelEvents; only a real wheel at the inner container's coordinates triggers their lazy load.

Three coordinate modes (priority): ref > (x, y) > viewport centre.

• Ref handling: same path as /click — refToLocator, falls back to refreshTabRefs with pre_wheel reason, throws StaleRefsError if still unresolvable
• Concurrency: wrapped in withTabLock
• Settle delays: 50 ms after move, 300 ms after wheel
• Plugin event: emits tab:mouse-wheel
• OpenAPI: annotated

2. /tabs/:tabId/init-script (authMiddleware)

POST /tabs/:tabId/init-script
{
  "userId": "...",
  "script": "window.__caps = []; const origFetch = window.fetch; window.fetch = ..."
}
→ { "ok": true, "scriptLen": 815 }

Wraps page.addInitScript({ content: script }). The script is evaluated in the page world before any other script on every navigation in the tab. Useful for hooks that must beat first-byte JS (e.g. install a fetch wrapper before the bundle imports it).

• Auth-gated (arbitrary JS in the page world)
• 256 KB body limit
• Single-script per call; multiple calls stack

3. /tabs/:tabId/capture-network (authMiddleware)

POST /tabs/:tabId/capture-network
{
  "userId": "...",
  "urlPattern": "graphql",    // optional, default "graphql" (regex, case-insensitive)
  "durationMs": 15000,        // capped at 60000
  "maxBodyBytes": 1000000,    // per capture
  "maxCaptures": 100
}
→ { "ok": true, "captureCount": 35, "captures": [ { "url", "status", "len", "body" }, ... ] }

Attaches page.on("response") for durationMs, then detaches and returns every matching response. Operates at the browser network layer, so it captures:

• Fetches from window.fetch references the page bundle cached before any in-page hook had a chance
• Fetches routed through a Service Worker
• XHR fetches that bypass monkey-patched prototypes

In practice this is the only reliable way to observe outgoing API traffic of SPAs that aggressively cache primitives at bundle init time.

• Auth-gated (response bodies may contain sensitive data)
• Per-capture and total-count caps prevent unbounded memory use

4. /tabs/:tabId/capture-requests (authMiddleware)

POST /tabs/:tabId/capture-requests
{
  "userId": "...",
  "urlPattern": "graphql",    // optional, default "graphql" (regex, case-insensitive)
  "durationMs": 15000,        // capped at 60000
  "maxBodyBytes": 200000,     // per capture
  "maxCaptures": 100,
  "includeHeaders": true      // optional, default true
}
→ {
    "ok": true,
    "captureCount": 36,
    "captures": [
      { "url", "method", "len", "body", "headers": { ... } },
      ...
    ]
  }

Symmetric counterpart to /capture-network: page.on("request") instead of page.on("response"). Returns POST body + headers for each matching request, captured at the browser network layer.

Motivation: /capture-network reveals what data the page receives, but is silent on what the page sends. Without the outgoing payload (CSRF tokens, doc IDs, pagination cursors, signed query params), it's impossible to replay or extend a captured GraphQL operation from outside the page. Hook-based approaches (window.fetch override, XMLHttpRequest.prototype.send override) fail against bundles that cache the original references at module init — verified empirically against Instagram's web client. Only a Playwright-level listener catches every outbound request.

• Auth-gated (request bodies / cookies / signed tokens are sensitive)
• headers returns the resolved request headers including cookie, x-csrftoken, x-fb-lsd, x-fb-friendly-name, etc. Set includeHeaders: false to omit them
• Per-capture and total-count caps prevent unbounded memory use
• Mirror code path of /capture-network (same handler shape, same lifecycle, same cleanup) for review symmetry

Verification

All four endpoints exercised on an Instagram DM thread that:

• Virtualises its message list (defeats /scroll's page-level wheel)
• Bundles React with a fetch reference captured at module init (defeats any in-page hook on either window.fetch or XMLHttpRequest.prototype)
• Uses a Service Worker for /api/graphql (defeats most network observers)

Results:

• /mouse-wheel at the container's centre coords with deltaY=-1500: scrollHeight grew from 1230 → 5382 in 8 batches (lazy load triggered)
• /init-script installed a fetch wrapper that captured a manual fetch("/api/graphql") call (verified hook installation), but missed all 35 of the page's own GraphQL fetches (confirms the closure-cache problem)
• /capture-network for 25 s while navigating to the thread: captured all 35 GraphQL responses (827 KB), including the one carrying the message list
• /capture-requests for 15 s while refreshing the thread: captured 36 GraphQL requests with full POST bodies + headers, including one IGDMessageListOffMsysQuery with its doc_id, fb_dtsg, lsd and pagination variables. The captured body was then used as a template — replacing only the variables.after cursor — to drive a full 14-batch pagination loop through the same /api/graphql endpoint, yielding 286 unique messages (the complete thread history). With only /capture-network, the same extraction would have stopped at the first 20 messages.

Also verified:

• node --check server.js passes
• All four endpoints return 400 on missing userId
• 404 on unknown tabId

[nayrosk]

Rebased onto the current master — no conflicts, and it now includes the recent fixes (postinstall env whitelist, VNC ENABLE_VNC override, BROWSER_IDLE_TIMEOUT_MS=0, and the fork-PR CI fix). The three commits add opt-in, low-level interaction endpoints (/tabs/:tabId/mouse-wheel, /init-script, /capture-network, /tabs/:tabId/capture-requests) and don't touch existing behavior.

This should merge cleanly now. Happy to split it into smaller PRs or address any review feedback — just let me know. Thanks!

[skyfallsin]

Thanks for the advanced interaction and capture endpoints. The feature set is useful, but this version needs changes before it can merge.

Please:

• add complete @openapi blocks for every new route and regenerate the committed openapi.json;
• track and await in-flight response-body reads before returning a network-capture response, so responses that arrive near the deadline are not silently omitted;
• use a bounded timeout for the mouse-wheel bounding-box lookup, consistent with the current click path;
• validate user-supplied regular expressions and numeric limits/durations as client input, returning 400 rather than an internal error for invalid values;
• rebase onto current master and add focused endpoint regression coverage.

Please keep the scope focused on these endpoints and their production contracts.

[nayrosk]

Thanks for the detailed review. All five points are addressed, rebased onto current master (v1.13.1) with no conflicts. Details below, in the order you raised them.

1. OpenAPI blocks + regenerated openapi.json

capture-network, capture-requests and init-script had no @openapi block at all, which was already failing tests/unit/openapi.test.js on three assertions (route coverage, and the committed spec matching a fresh regeneration). All three now have complete blocks, and the mouse-wheel one is fleshed out with its 422 and the Error schema $refs that the rest of the API uses.

• security: BearerAuth is now declared on the three authMiddleware() routes, matching what the code actually enforces
• Added a Network tag in lib/openapi.js for the two capture endpoints, since neither Interaction nor Content really describes them
• Documented status is 403, not 401: that is what requireAuth returns on a missing token (lib/auth.js:85), and the endpoint tests now assert it
• openapi.json regenerated, openapi.test.js back to 16/16

2. In-flight response body reads

Real bug, thanks for catching it. response.text() is async, so a response landing near the deadline still had its body in flight when the listener was detached and the reply was sent, and it was dropped silently.

Capture logic moved to lib/network-capture.js:

• Every started body read is tracked, and the pending ones are awaited before returning, under a bounded 5 s grace period so a hung read cannot extend the request indefinitely
• Capture slots are reserved synchronously when the event fires, so arrival order and the maxCaptures budget no longer depend on which read finishes first
• Listeners are detached in a finally block, so an error mid-window cannot leak one
• A body that still has not arrived after the grace period comes back as __BODY_PENDING__ and is counted in the new pendingBodies field, rather than being dropped without a trace
• Added droppedByLimit so a caller can tell "nothing else matched" from "hit the cap"

While writing the tests I also found that a failed read was being truncated to maxBodyBytes, which mangled the __BODY_ERROR__ reason. Truncation now applies to bodies only.

3. Bounded bounding box lookup

mouse-wheel now uses the exact idiom from the current click path: Math.max(500, Math.min(3000, remainingBudget())) with a 422 on detach, instead of inheriting Playwright's 30 s default and burning the whole handler budget into a 500. The pre-wheel refs refresh uses the same budget, the way /click does.

One related fix: the catch around that refresh was testing for pre_click_refs_timeout, but refreshTabRefs builds its label from the reason, so it emits pre_wheel_refs_timeout. The guard never matched and always rethrew.

4. Client input validation

New lib/interaction-params.js, imported by the routes themselves so the tests exercise the real guards instead of a copy that drifts (the viewport guard currently lives in two places).

• urlPattern is compiled in a try/catch and returns 400 with the syntax error. Previously an invalid regex reached sendError() as a 500
• 200 character cap, plus a guard against nested quantifiers like (a+)+, since the pattern is tested against every URL the browser sees
• durationMs 100..60000, maxCaptures 1..500, maxBodyBytes 1024..5000000, all rejected with 400 rather than silently clamped. A caller asking for durationMs: 600000 now learns the cap instead of quietly getting 60 s
• mouse-wheel deltas and coordinates must be finite and in range, x and y must be provided together
• Every range is documented in the spec with minimum/maximum

5. Rebase + regression coverage

Rebased onto master at v1.13.1, no conflicts. Three new test files, 104 tests:

• tests/unit/interactionParams.test.js: the validators, including invalid regexes, out of range limits and the delta/coordinate rules
• tests/unit/networkCapture.test.js: the capture helpers against a fake page, covering the in-flight drain, the bounded drain fallback, arrival ordering when bodies resolve out of order, maxCaptures, truncation and failed reads
• tests/unit/interactionEndpoints.test.js: endpoint level, asserting 400/403/404 against a live server. Every case is rejected before the tab lookup, so it needs no browser

Full unit suite is 891 passed. The 5 browser-dependent suites (cookies, security, tabLifecycleContract, tabRecycling, operationalFailures) fail in my environment for lack of a local camoufox binary, and fail identically on a pristine master checkout, so nothing there is new. tests/e2e and tests/live are untested locally for the same reason and will need CI.

Scope is unchanged, still just these four endpoints and their contracts. Happy to split the capture endpoints out into their own PR if that reviews more easily, just let me know. Thanks!

@skyfallsin