Skip to content

feat(snapshot): add ref annotation bounding boxes to snapshot response - #5724

Open
melxusgid wants to merge 1 commit into
jo-inc:masterfrom
melxusgid:feat/screenshot-ref-annotations
Open

feat(snapshot): add ref annotation bounding boxes to snapshot response#5724
melxusgid wants to merge 1 commit into
jo-inc:masterfrom
melxusgid:feat/screenshot-ref-annotations

Conversation

@melxusgid

Copy link
Copy Markdown

Feature

Adds annotateScreenshot=true query parameter to GET /tabs/{tabId}/snapshot.

When set, the response includes a screenshotAnnotations array with element bounding boxes:

{
  "screenshotAnnotations": [
    { "refId": "e1", "role": "button", "name": "Search", "x": 120, "y": 45, "width": 200, "height": 40 },
    { "refId": "e2", "role": "link", "name": "Sign in", "x": 800, "y": 20, "width": 80, "height": 24 }
  ]
}

This makes human-in-the-loop workflows possible: clients can overlay ref IDs on the screenshot image.

Implementation

New helper getRefAnnotations(page, refs) resolves each ref to its bounding box using the same refToLocator logic already used by click/type endpoints. Skips elements that are no longer visible.

Works in all three snapshot paths:

  • Cached offset chunks
  • Google SERP fast path
  • Full ariaSnapshot

No new dependencies. Pure Playwright boundingBox() calls.

Fixes #5086

Adds a new query parameter annotateScreenshot=true to the
GET /tabs/{tabId}/snapshot endpoint. When set along with
includeScreenshot=true, the response includes a
screenshotAnnotations array with {refId, role, name, x, y, width, height}
for each accessible element ref that has a bounding box on the page.

This makes human-in-the-loop workflows possible by letting clients
overlay ref IDs on the screenshot image.

New helper function getRefAnnotations(page, refs) resolves each ref
to its Playwright bounding box using the same refToLocator logic
used by click/type endpoints.

Works in all three snapshot paths: cached offset, Google SERP,
and full ariaSnapshot.

Fixes jo-inc#5086
@skyfallsin

Copy link
Copy Markdown
Contributor

thanks for this—the human-in-the-loop motivation makes sense. I’m thinking a little more about the right API boundary here, particularly annotated images versus returning geometry without making the capture path brittle. stay tuned.

@skyfallsin skyfallsin added the considering Under consideration; no decision yet label Jul 20, 2026
@skyfallsin

Copy link
Copy Markdown
Contributor

Thanks for this feature work. We do not want to extend the core snapshot response with screenshot-ref annotation rendering. Please move this into a plugin or a separate plugin-owned endpoint, so the annotation implementation and its screenshot contract can evolve without coupling to the core snapshot/reference pipeline.

For a reworked version, please also:

  • use an explicit, request-budget-aware timeout for every bounding-box lookup; current core code does not permit unbounded locator.boundingBox() calls;
  • avoid serial unbounded work across a large reference table;
  • document the plugin endpoint contract and add focused coverage for ref-to-box alignment, hidden elements, iframe refs, and timeout behavior.

Please re-open or update this PR with the plugin design rather than adding the feature to the core snapshot route.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

considering Under consideration; no decision yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Overlay element ref IDs on screenshots

2 participants