feat(scroll): find off-screen targets in one command with --until

[thymikee]

Why

User feedback from a side-by-side run against Claude's built-in iOS tool: agent-device won, and the one recurring complaint was that the agent "scrolls too little to find an element", called out as the agent's fault rather than the tool's.

Investigating that turned up three things, and the agent was the least of them.

Our own guidance taught the slow loop. The skill's routing card and the MCP server guide both prescribe bare scroll <direction> --settle. The off-screen refusal hint went further and told the agent to scroll "in small steps", warning that a single large scroll overshoots. The agent followed our instructions exactly.

scroll bottom already existed and nothing pointed at it. It runs a scroll-and-check loop server-side in one request. The string appeared in exactly one agent-visible place: the command's own description. Not in the skill, not in the MCP guide, not in the snapshot legend that prints [off-screen below], not in the refusal hint.

A larger amount buys almost nothing, and said so dishonestly. One gesture is clamped to the viewport axis minus 10% padding at each end, in both the TypeScript planner and the Swift runner. Default travel is 0.65 of the viewport on iOS, 0.6 elsewhere, and the ceiling is 0.8. So scroll down 3 does not scroll three viewports, it scrolls 0.8 of one, and the success message reported "Scrolled down by 3". The dogfood help topic used scroll down 3 as its worked example.

What this changes

scroll <direction> --until <selector>. Repeats scroll-and-check passes until the selector matches a node that is on screen, then stops. One request instead of the scroll-then-look-again loop, and because it checks between passes it stops on the target rather than sailing past it, which retires the accuracy-versus-distance tradeoff the old hint was working around.

An unreadable capture is refused before either check runs, as scroll_until_capture_unreadable with a no-capture or sparse-tree cause. Coercing a missing tree to [] would otherwise reach the edge analyzer as "no room below", so a failed read would report end-of-content.

It stops three ways. Matched is the success. Running out of scrollable content and spending the 12-pass budget are separate typed failures (scroll_until_edge_reached, scroll_until_pass_limit) with different hints, because the corrective action differs. End-of-content reuses the same signal scroll top/scroll bottom already trust, so both stop in the same place. Horizontal scrolls have no such analyzer and are bounded by the budget alone. --until is refused on the top/bottom directions, which already carry a stop condition, rather than letting one silently win.

Honest distance reporting. The message now names the travel the planner honored: Scrolled down by 3 of the viewport (700px). The result JSON already carried the honored pixels; only the prose lied.

Web amount was in the wrong unit. amount is a viewport fraction everywhere else, and the browser backend passed it straight through as CSS pixels, so scroll down 0.5 travelled half a pixel. It now scales the default step the way the Linux pointer backend scales its wheel clicks.

Guidance. The skill, the MCP server guide, the snapshot legend, the workflow card, the gestures topic, and both off-screen refusal hints now name the one-command path. The dogfood example is no longer a number that does nothing.

Shape

scroll reaches a device in exactly one place (ADR 0019, the daemon's generic route), so the feature lives in one module beside it: src/daemon/scroll-until.ts owns the pass loop, the arrival check, the capture-readability check and both failure shapes. No new package surface, no shared-module indirection, and nothing to keep two callers in agreement.

The until plan tier admits on the same facts the edge tier does, so it shares that use declaration rather than duplicating one. The end-of-content signal is the one scroll top/scroll bottom already trust, reached through a small canScrollFurtherAtEdge export on the existing scroll-edge-state subpath.

scroll moved out of gestures.ts into its own runtime module first, as a pure move with its tests, since that file was already past the threshold the repo sets for adding behavior.

Validation

Live on an iOS Simulator against Settings, using the built CLI on this branch, reaching the Developer row from the top of the list. Three runs each.

	Commands	Wall clock
scroll down + snapshot -i, repeated until found	4	2.37s / 2.41s / 2.53s
scroll down --until 'label=Developer'	1	0.72s / 0.75s / 0.76s

The command count is the part that matters more than the clock: the old path also makes the agent read two full snapshots it does not need.

Web amount verified in a live browser against Wikipedia, reading window.scrollY between commands and resetting to the top each time:

Command	Observed travel
scroll down	300px
scroll down 0.6	300px
scroll down 0.5	250px
scroll down 1.2	600px
scroll down --pixels 250	250px
scroll down --pixels 250 --duration-ms 300	250px
scroll down 1.2 --duration-ms 300	600px

Also confirmed live on this branch: an already-visible target costs zero gestures, the edge rejection prints its hint, a missing selector reports end-of-content after 2 passes rather than burning the 12-pass budget, and scroll down 3 now reports Scrolled down by 3 of the viewport (700px) on an 874-point viewport.

Gates, all green: unit (9442 tests), provider-integration (199), typecheck, lint, format, layering, fallow, di-seams, integration-progress, build, package, replay-compat, daemon-wire-compat, mcp-metadata, command-docs, agent-guidance, gate-manifest, depgraph, bundle-owner-files, freerange, fixture-cache, production-exports.

production-exports passes but prints 62 repo-wide unused exports as advisory output. None of them come from this branch, checked against its JSON.

[github-actions]

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	4.53 MB	4.54 MB	+3.4 kB
Package (unpacked)	4.53 MB	4.54 MB	+3.4 kB
Package (download)	1.34 MB	1.35 MB	+1.4 kB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	27.5 ms	27.4 ms	-0.0 ms
CLI --help	76.5 ms	76.3 ms	-0.2 ms

[thymikee]

The new loop must distinguish failed observation from end of content. Both captureUntilNodes and captureRuntimeScrollNodes discard capture quality and turn missing nodes into an empty array; the vertical edge analyzer then reports no room, producing scroll_until_edge_reached without a readable capture. Preserve the capture failure before matching or edge detection, and add regressions through both execution routes for missing/sparse captures.

The web amount conversion also needs a live-browser check: verify observed scrolling for a relative amount and an explicit pixel distance, including the paced path. The reported iOS run is useful, but does not cover this separate behavior change.

Please itemize the 7.7 kB unpacked growth and explain whether more of the existing scroll-pass machinery can be reused, or why a smaller design was rejected. The iOS CI failure happened while waiting for the test app before scrolling, so it looks unrelated to --until rather than evidence against the local measurements.

[thymikee]

All four addressed. Rebased onto main; the branch is linear on 6ca66c9 now.

Failed observation vs end of content

You were right, and it was worse than a reporting problem. Both captureUntilNodes and captureRuntimeScrollNodes coerced a missing tree to [], and analyzeScrollEdgeState reads an empty tree as "no room below", so a capture that failed produced scroll_until_edge_reached — the loop concluded something about the content from a screen nobody could read.

Both routes now classify the capture before the selector match and before the edge analyzer, and refuse rather than continue. The classifier is scrollUntilCaptureRefusal in @agent-device/selectors/scroll-until-match, next to the visibility predicate, so both routes ask one question:

• no tree, or an empty tree → no-capture
• the backend's own snapshotQuality.state === 'sparse' → sparse-tree
• the legacy iOS application-root-only shape → sparse-tree

Sparseness reuses the signals absence assertions already trust (isLegacySparseIosInteractiveSnapshot and the quality verdict) rather than adding a second definition of readable. The failure is scroll_until_capture_unreadable with a captureRefusal field, and its hint points at the recovery the shape calls for: snapshot -i for a missing tree, screenshot and coordinates for a sparse one.

Truncation is deliberately not refused. A truncated tree is real and readable with its tail missing, and refusing it would fail large screens where the target is plainly in view. There is a test pinning that.

Regressions, both routes and both shapes: scroll.test.ts and scroll-runtime.test.ts each cover a missing capture and a sparse one, and scroll-until-match.test.ts covers the classifier directly including the healthy, recovered and untagged cases that must not be refused.

Chasing this also turned up a bug in my own error builders: the typed fields were nested one level too deep, so selector, direction and passes were unreachable on the two --until failures I had already shipped in this PR. Flattened.

Web amount, live browser

Fair — the iOS run said nothing about this path. Wikipedia through the managed backend, reading window.scrollY between commands, resetting to the top before each:

Command	Observed travel
scroll down	300px
scroll down 0.6	300px
scroll down 0.5	250px
scroll down 1.2	600px
scroll down --pixels 250	250px
scroll down --pixels 250 --duration-ms 300	250px
scroll down 0.6 --duration-ms 300	300px
scroll down 1.2 --duration-ms 300	600px

The paced path is covered on both branches, and the split sums to the requested distance rather than drifting. scroll down 0.5 is the case that used to travel half a pixel.

The 7.7 kB

Measured as an A/B: origin/main built in a scratch worktree with the same gate command, diffed file by file against this branch. JS only, .d.ts excluded:

	Bytes
selector-pipeline.js (new chunk)	+5,902
snapshot-occlusion.js (new chunk)	+3,149
absence-observation.js (new chunk)	+1,489
selector-pipeline-policy.js (new chunk)	+1,308
android-input-ownership.js	−3,136
runtime.js	−2,036
app-inventory-contract.js	−1,184
internal/daemon.js	+1,390
cli-help.js	+816
registry.js	+608
scroll-command.js	+301
server.js	+227
everything else	+543
net	+9,377

Two things carry it, and new logic is the smaller one.

About 3.8 kB is genuinely new: the daemon's until executor plus the capture refusal, the flag through the registry, the compatibility assertion, and the help and guidance text. The loop itself is roughly thirty lines.

The rest is that scroll now reaches the selector pipeline at all. Scroll had never resolved a selector before this PR, so selector-pipeline, its policy table, and snapshot-occlusion enter its reachable set for the first time. Because I put those behind function-scoped await import calls, the bundler emits them as standalone chunks instead of inlining them, which is why four chunks appear and three shrink.

Reuse, and the smaller designs

Reused rather than reimplemented: the edge signal (analyzeScrollEdgeState, the same one scroll top/scroll bottom stop on), the pass-loop shape, the wait pipeline row and the visibility resolver, the sparse-capture predicate, and the runtime use declaration — --until admits on the same facts the edge tier does, so it shares that declaration rather than duplicating it. The registry caught the duplicate the first time I tried to add one.

Two smaller designs I did try and back out of:

Static imports instead of lazy. Measured at roughly 1.5 kB smaller, because nothing gets split into its own chunk. It makes scroll-until-match a 66-module eager entry surface, over the domain-facade ceiling of 20, which needs an APPROVED_OVER_CEILING row. I took the bytes over the approval row and the startup cost; Bundle Size reports startup unchanged to slightly better on this branch.

Extracting the sparse predicate into its own module so both callers get it without the ad-script closure behind absence-observation. That is arguably the cleanest shape, but the no-growth ratchet counts modules, and a new file adds one to every closure that reaches it — four entries regressed by exactly one. Reverted in favour of a lazy import.

The iOS CI failure

Agreed, and confirmed. It failed at step 7 in observeFixtureHome, a wait text that returned wait_capture_stalled with readableCaptures: 0 after a single 10 s poll. That is the app-mount capture-stall class, and the scenario reaches no scroll at all before it. No --until code runs on that path.

Worth noting the shape: readableCaptures: 0 is the same distinction wait already draws and the one this PR was missing.

[thymikee]

The new refusal check at 17956a5 still misses the backend's sparse verdict. BackendSnapshotResult carries it as quality, while scrollUntilCaptureRefusal reads snapshotQuality; the daemon test uses that wrong field too. The in-process route can also drop top-level quality when it selects result.snapshot. Preserve the canonical quality annotation on both routes and test real backend-shaped sparse results, proving they fail before matching, edge analysis or scrolling. The added web measurements and size explanation address those earlier requests.

[thymikee]

Right on both counts, and the daemon test was the worse half: it asserted through snapshotQuality against a route that receives quality, so it passed while the real field went unread. A green test proving nothing.

Fixed at 70c7b96, rebased onto bd42b26.

The classifier now reads the verdict under every spelling a capture can arrive in. SnapshotState says snapshotQuality, BackendSnapshotResult says quality, and a backend result can carry a nested state as well. scrollUntilCaptureRefusal resolves all three through one canonicalCapture step: the nested state wins on nodes and backend, and the verdict is taken from whichever level carries one. The top-level payload goes through readSnapshotQualityVerdict, so a malformed object falls through to the legacy node-shape detector instead of being silently treated as a verdict.

Asking each route to normalize first is exactly what produced this bug, so the one place that asks the question is the place that understands the shapes.

The in-process route no longer picks a level. It was doing result.snapshot ?? result, which drops a top-level quality whenever a nested snapshot exists. It now hands the whole result to the classifier and reads nodes from either level.

Tests use real backend shapes and prove the ordering. Each route now covers a missing capture and a sparse one, with a gesture spy asserting zero scrolls, and the sparse fixtures deliberately have content below the fold so an edge verdict would be wrong there too:

• daemon route: the owner's quality spelling, the shape resolveBoundScrollRuntime actually receives
• in-process route: top-level quality beside a nested snapshot, and a verdict carried on the nested snapshot itself
• classifier: all four arrangements, plus a malformed payload that must not be mistaken for a verdict

Live re-check on the simulator after the change, since the classifier now runs against real captures and a wrong field would refuse everything: scroll down --until 'label=Developer' still lands in one pass, and a missing selector still reports end-of-content rather than a capture failure.

[thymikee]

Due-diligence pass on the design. A large part of it was not needed, and the reason is worth stating plainly: I designed for two live execution routes without proving both were reachable.

The in-process scroll route is dead code

I built --until into the daemon's generic route and into the in-process command runtime, on the premise that both run in production and must not disagree. Only the daemon route runs.

Proved empirically rather than by reading. I put a tripwire throw at the top of the in-process scrollCommand, rebuilt, and ran:

• the whole provider-integration suite, 63 files and 199 tests — zero hits
• every scroll shape live on a simulator (down, an explicit amount, top, bottom, --until, --settle) — zero hits, all succeeded
• the unit suite — exactly one file reached it: scroll.test.ts, its own test

The public typed client's interactions.scroll returns CommandRequestResult; it sends a daemon request. Nothing binds the in-process one.

So the "two paths must agree" justification, and everything built to serve it, was scaffolding for a caller that does not exist.

What that removed

The in-process --until implementation and its tests. Both new package subpaths — capture-kit/scroll-until-visible and selectors/scroll-until-match — collapse into src/daemon/scroll-until.ts, beside the route that runs it. With no package entry surface there is no domain-facade ceiling to satisfy, so all four lazy imports are gone, and with them the chunk-splitting they caused and the layering-gate registrations they needed. The classifier, the pass loop and the message builders were three pieces across two packages for one caller; they are one file now.

	Before	After
Bundle growth	+9,377 B	+6,499 B
New chunks emitted	4	0
New package subpaths	2	0
Lazy-import workarounds	4	0
Insertions	2,213	1,724

Every remaining byte lands in the daemon bundle where the feature lives, instead of duplicating shared modules into standalone chunks. The earlier itemization is obsolete: there is no longer a chunk-splitting line, and the growth is the daemon route (+3.8 kB, the loop and the selector pipeline it now reaches), help and guidance text (+0.8 kB), the registry tier and flag (+0.6 kB), the compatibility assertion (+0.3 kB), the MCP guide (+0.2 kB) and the web conversion (+0.1 kB).

What stayed, and why

The until plan tier still shares the edge tier's use declaration rather than duplicating it. The capture-readability check still reads the verdict under every spelling, since a backend result can nest a state — that was your last finding and it is unchanged. The end-of-content signal is still the one scroll top/scroll bottom trust, now reached through a small canScrollFurtherAtEdge export on the existing scroll-edge-state subpath rather than a new surface.

The scroll extraction out of gestures.ts stays. That file was already past the 500-line threshold the repo sets for adding behavior, and the honored-travel change adds some.

Verification

Unit 9445, provider-integration 199, and typecheck, lint, format, layering, fallow, di-seams, integration-progress, mcp-metadata, build, package, production-exports, bundle-owner-files, command-docs, depgraph all green. Re-verified live on the simulator after the rework: one-pass arrival, the already-visible no-op, end-of-content after two passes, the edge rejection, and the clamped-amount message.

Separate finding

scrollCommand in the in-process runtime is reachable only from its own test, and that predates this PR. Worth its own issue rather than widening this one.

[thymikee]

The quality-field fix and single daemon implementation look sound at 09320ad. One regression fixture still needs correcting: climbingHierarchy calls the stateful targetTop() twice for one rectangle, producing inverted bounds on the first capture and advancing the target twice. Read it once, use that value for both bounds, and assert the intended arrival pass count so the test proves repeated scrolling on valid geometry. Also remove the duplicated edge-rejection, unavailable-capture and missing-capture tests left in scroll-runtime.test.ts. The reported live evidence and revised size explanation address the earlier gaps.

[thymikee]

Dug for second-order size wins. Found a small one, and found that my earlier attribution was wrong. Correcting that first, because it changes what is worth cutting.

The selector pipeline costs 35 bytes, not 3.8 kB

I claimed the growth was mostly scroll reaching the selector pipeline for the first time. That was reasoning from the import graph, not measurement. Stubbing the pipeline call and rebuilding moves the bundle by 35 bytes: the daemon already carries it for wait, is and find.

Same for the rest of the module's dependencies, each measured by stubbing it out and rebuilding:

Dependency	Cost
the edge signal	266 B
the legacy sparse predicate	189 B
the visibility resolver	57 B
the quality-verdict reader	15 B
the selector pipeline	35 B

Everything the feature needs was already in the daemon bundle. There is no dependency to remove.

Where the bytes actually are

Stubbing the whole --until arm puts the loop, the classifier, the messages and the hints at 3,719 B, of which about 1,400 is hint and message string literals. The remaining ~2,500 B of the total is help and guidance text, the registry tier and flag, the compatibility assertion, the MCP schema entry, and the web conversion.

Net across the branch, user-facing prose is about 2,400 B — roughly 40% of the growth, and it is the part that fixes the behavior the original report was about.

What I trimmed

Prose that was stated more than once on the same surface. The 0.8 cap appeared in the gestures topic, the dogfood example and the command description; the top/bottom rule appeared in both the command description and the until field description, which the MCP schema shows together. Two dead exports went too: ScrollUntilCaptureRefusal and ScrollUntilOutcome were exported and used nowhere outside their module.

+6,499 → +6,252 B. Modest, and it is what is left once the fat is gone.

What I looked at and did not cut

The nested-snapshot branch in the capture reader. A live iOS run never hits it, but a one-platform sample is not proof. I asked the type system instead: the runtime capture result does carry .snapshot, so the branch is reachable by contract. Deleting it on one run would repeat the mistake that produced the two-route design.

The error hints. They are the longest literals, and they are what an agent acts on instead of re-planning. Tightened the wording, kept every action.

assertScrollUntilCompatible. 301 B including its message. It is a real guard against a request with two stop conditions, and making it unrepresentable at the grammar layer would still need the message.

My read is that the remaining growth is the feature, not overhead. Gates green, and re-verified live after the trims: one-pass arrival, end-of-content after two passes, the edge rejection, and the clamped-amount message.

[thymikee]

Both correct. Fixed at d34bfeb, rebased onto main.

The climbing fixture was broken. climbingHierarchy interpolated targetTop() once per bound, and the generator is stateful, so each rectangle read two different positions: the first capture produced [24,1400][374,854], a bottom above its top, and the row advanced two screens per capture instead of one. It now reads the position once and uses it for both bounds.

The test now asserts the arrival count. The row starts two screens below the fold and climbs one per capture, so it arrives on the third: exactly two gestures. assert.equal(result.passes, 2) plus the matching message, instead of the >= 1 floor that could not tell repeated scrolling from a lucky first capture. Confirmed stable across three consecutive runs.

Removed the duplicated cases in scroll-runtime.test.ts. My earlier splice appended new versions without removing the originals, leaving four tests defined twice under identical names. That file now keeps only what is route-level and cannot be reached from the module: the result envelope, the parse rejection on an edge direction, and the admission refusal when the owner declares no capture. Arrival, end-of-content, the pass budget and every capture refusal are covered against the module in scroll-until.test.ts, which is where the behavior lives. 21 tests down to 13.

One note on my own verification. A first pass reported an unrelated Android gesture-admission failure, both cases timing out at 5000 ms. Other worktrees on this machine were running suites concurrently; alone it passes. Flagging it so the run is not mistaken for a signal from this branch.

[thymikee]

The fixture and duplicate-test findings are addressed at d34bfeb. The test now uses valid bounds and proves two scroll passes; no remaining code findings, and the reported iOS/browser checks cover the changed behavior. The red iOS job timed out compiling the unchanged snapshot-bridge test before scrolling; it looks unrelated and needs a green rerun before merge.

[thymikee]

All four blockers fixed, the safe drops applied, the nit taken. Rebased onto main at 52665cf. Net −743 lines.

Blockers

1. The command-runtime executor is gone, not moved. You are right that it was not a pure move, and right that it was dead: src/sdk/index.ts exports only the client, nothing calls createAgentDevice().interactions.scroll, and the daemon dispatches scroll only through resolveBoundScrollRuntime. Deleted scroll.ts, scroll.test.ts, the scroll slot in InteractionCommands/BoundInteractionCommands and the parity assertion. 559 lines. The fallow gate then caught the follow-on: requireResolvedPoint was exported only for that file, so it is local again.

That also retires the disagreement you named — there is now one scroll executor, and it knows --until.

2. ScrollUntilCapture is gone. capture is typed () => Promise<SnapshotResult>. That deleted the unknown, the readSnapshotQualityVerdict import, canonicalCapture and the four-arrangement test. You were right about the provenance: result.snapshotQuality was never a production shape, it came from my in-process route.

captureRefusal no longer re-implements the sparse rule. sparseQualityForSnapshot is now exported as sparseCaptureQuality and both callers use it, so "too sparse to trust" has one definition.

3. --until now shares is visible's predicate. evaluateIsPredicate({ predicate: 'visible' }) instead of the bare isVisibleOnScreen, so the Android visibleToUser rule, non-positive rects, the hittable fallback and anchor resolution all apply. Verified live that the two agree: scroll down --until 'label=Developer' then is visible 'label=Developer' passes on the same node.

4. Contract and schema aligned. ScrollCommandResult gains until?: string, and the passes doc now says "Edge and until scrolls only" to match the schema that derives from it.

Safe drops

honoredScrollPixels now lives beside honoredScrollDurationMs in the contract, one copy. The Linux backend scales by DEFAULT_SCROLL_AMOUNT instead of a hardcoded 0.6, which is what the export was for. scrollEdgeUnsupported and scrollUntilUnsupported are one scrollCaptureUnsupported taking the subject. resolveBoundScrollRuntime builds the stop condition once and passes it to both the assertion and the plan. The edge-loop hint no longer prescribes the manual scroll-and-snapshot loop this PR retires. The packages/selectors/package.json ellipsis change is reverted. The stale test comment is gone.

Nit

formatScrollEdgeMessage takes a params object.

The shared loop

Agreed, and agreed it is not this PR. Filed as #2468 with your framing: one runScrollPasses({ shouldStop, passLimit }), scroll bottom as a stop condition, and the capture refusal landing once for both callers. I confirmed the defect you spotted — scroll bottom on an unreadable capture reports "Already at bottom" — and left it out of here because it changes the failure surface of a different command.

Verification

Unit 9474, provider-integration 203, and typecheck, lint, format, layering, fallow, di-seams, integration-progress, mcp-metadata, build, package, production-exports, command-docs, bundle-owner-files, depgraph. Re-verified live on a simulator after the predicate change: one-pass arrival, the already-visible no-op, end-of-content after two passes, the edge rejection, the clamped-amount message, and scroll bottom still reaching the edge.

[thymikee]

Looks good at 52665cf. The duplicate executor is gone, capture quality uses the shared sparse check, and --until uses the same visibility predicate as is visible. No remaining code blockers; the reported simulator/browser evidence covers the changed paths, and the shared edge-loop follow-up stays in #2468.

All completed checks pass; iOS smoke is still running. One nonblocking cleanup: refresh the PR body to reflect the single executor and latest validation counts, since its earlier implementation description is now stale.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-10 15:13 UTC