Skip to content

docs(ego-browser): document task space reassignment and screenshot coordinate scaling - #257

Open
liuhedev wants to merge 1 commit into
citrolabs:mainfrom
liuhedev:docs/skill-task-space-and-screenshot-coords
Open

docs(ego-browser): document task space reassignment and screenshot coordinate scaling#257
liuhedev wants to merge 1 commit into
citrolabs:mainfrom
liuhedev:docs/skill-task-space-and-screenshot-coords

Conversation

@liuhedev

Copy link
Copy Markdown

What

Adds two operational caveats to skills/ego-browser/SKILL.md:

  1. In the Task spaces section: a task space can vanish or get reassigned without warning when another process shares the same browser instance (e.g. a second concurrent ego-browser session), causing a later useOrCreateTaskSpace(id) call to throw task space not found: <id>. Documents the recovery pattern — treat it as a cold start, re-navigate, and re-verify state rather than assuming prior context still holds.
  2. In the Caveats section: captureScreenshot() images can be scaled down from the real viewport (e.g. a page rendered smaller than its actual pixel size), so clicking at coordinates measured directly off the screenshot image lands off-target. Documents checking image dimensions against pageInfo()'s pw/ph and multiplying accordingly, plus the getBoundingClientRect() alternative that skips the conversion entirely.

Why

Both were hit during a real multi-hour agent session doing DataArts Studio console operations (navigating, clicking, running scripts across many heredoc rounds). The task space was silently replaced mid-session (likely because another concurrent ego-browser session shared the same browser instance), and several clicks landed on the wrong element because coordinates were read directly off a screenshot without accounting for its scale factor relative to the real viewport. Neither gotcha was documented, so the agent spent significant time debugging both from scratch. Documenting them should save that debugging time for future sessions.

How to verify

Documentation-only change to skills/ego-browser/SKILL.md, no code/runtime changes. Read the two new paragraphs in context (Task spaces section, and Caveats section) to confirm they fit the surrounding tone and don't duplicate existing guidance.

Impact

  • Does not touch the helper surface (useOrCreateTaskSpace, captureScreenshot, click, etc.) — no runtime behavior changes.
  • No SKILL.zh.md exists yet in this repo, so no i18n sync needed.
  • No code touched, so npm test / npm run typecheck / npm run validate:site-skills are unaffected by this change.

…ordinate scaling

Two operational gotchas discovered in real-world multi-session usage:

- A task space can vanish or get reassigned without warning when another
  process shares the same browser instance, causing a later
  useOrCreateTaskSpace(id) call to throw "task space not found". Documents
  the recovery pattern: treat it as a cold start, not a retryable error.
- captureScreenshot() images can be scaled down from the real viewport,
  so clicking at coordinates measured directly off the screenshot lands
  off-target. Documents checking image dimensions against pageInfo()'s
  pw/ph and the getBoundingClientRect() alternative.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant