Skip to content

feat(ios): drive ASWebAuthenticationSession sign-in sheets in place (#2438) - #2448

Open
thymikee wants to merge 8 commits into
mainfrom
claude/agent-device-issue-2438-b5f950
Open

feat(ios): drive ASWebAuthenticationSession sign-in sheets in place (#2438)#2448
thymikee wants to merge 8 commits into
mainfrom
claude/agent-device-issue-2438-b5f950

Conversation

@thymikee

@thymikee thymikee commented Sep 10, 2026

Copy link
Copy Markdown
Member

Summary

iOS apps that sign in via ASWebAuthenticationSession present the identity provider in com.apple.SafariViewService, out of the app's process. Two facts, both verified live on the iOS 26.2 Simulator, made these flows unautomatable: activating or launching the host cancels the auth session, and the host AX bridge cannot see the sheet because the app stays the AX primaryApp.

This serves and drives the sheet in place:

  • A closed registry names the host (contracts/fixtures/ios-system-surface-hosts.json, mirrored by the TypeScript and Swift registries under a parity test).
  • The runner reads and drives the host without activation and never adopts it as the session target, so the session stays bound to the app.
  • The Simulator route detects a running host with a cheap device-scoped ps probe and takes the runner path; the bridge would serve the occluded app tree as if healthy.
  • open com.apple.SafariViewService is refused (UNSUPPORTED_OPERATION); captures carry a system-surface disclosure.

Presence is foreground state, not tree content — a torn-down host serves a richer tree than a live one — and the never-activate guard is what keeps that predicate sound, which also makes issue #2438's stale-tree bug unrepresentable for this flow. Design rationale in the ADR 0004 amendment.

17 files, Apple platform + contracts + docs only. Closes #2438.

Validation

Tested at 0cde7a4d2e. pnpm check:affected --run passes (40/40; lint, typecheck, layering, fallow, unit). Swift: runner builds + 2 registry parity tests pass.

Live on iPhone 17 / iOS 26.2 (real daemon + runner, an ASWebAuthenticationSession probe app + local IdP): open svs refused with the sheet intact; wait label="LOGIN"press role=textfield label="Email" + typefill role=securetextfield (values confirmed) → press role=button label="LOGIN" fired the auth callback and returned to the app; post-dismiss captures revert to the bridge with no disclosure.

Follow-ups: #2449 (test-app fixture + CI lane), #2450 (verify/settle lineage guard), #2451 (selector-route disclosure parity), #2452 (ADR 0011 offscreen comment).

Caveat: the first snapshot after a cold runner can return 0 nodes (pre-existing mid-bundle flake); use wait stable/wait <element>.

…2438)

iOS apps that sign in via ASWebAuthenticationSession present the identity
provider in com.apple.SafariViewService, out of the app's process. Two facts,
both verified live on the iOS 26.2 Simulator, made these flows unautomatable:
activating or launching the host cancels the auth session, and the host AX
bridge cannot see the sheet because the app stays the AX primaryApp.

Serve and drive the sheet in place. A closed registry names the host (shared by
the TypeScript and Swift sides under a parity test); the runner reads and drives
it without activation and never adopts it as the session target; and the
Simulator route detects a running host with a cheap device-scoped ps probe and
takes the runner path, since the bridge would serve the occluded app tree as if
healthy. open refuses to launch a registered host, and captures carry a
system-surface disclosure.

Presence is foreground state, not tree content: a torn-down host serves a richer
tree than a live one, so content heuristics cannot tell them apart. The
never-activate guard is what keeps the foreground predicate sound, which also
makes the stale-tree failure mode unrepresentable for this flow.

Closes #2438
@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.53 MB 4.55 MB +14.8 kB
Package (unpacked) 4.53 MB 4.54 MB +14.8 kB
Package (download) 1.34 MB 1.35 MB +3.9 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 26.1 ms 25.3 ms -0.8 ms
CLI --help 70.9 ms 68.2 ms -2.7 ms

@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstack.github.io/agent-device/pr-preview/pr-2448/

Built to branch gh-pages at 2026-09-10 19:43 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@thymikee

Copy link
Copy Markdown
Member Author

There are still correctness gaps at 0cde7a4.

The presence probe caches absence for one second and treats ps failures as absence. A sheet opened just after an app capture, or while ps fails, therefore falls through to the bridge and returns the occluded app tree. Cache only positive presence, route unknown results to the runner, and test absent-to-present within the TTL plus probe failure.

The explicit-URL branches in openIosApp return before the new host guard. open com.apple.SafariViewService https://example.com can still launch the host and cancel authentication. Apply the refusal before every resolved-host launch or termination, and test the URL variants.

The provenance is reduced to a warning instead of reaching comparison identity. App and sheet captures can therefore enter legacy same-presentation matching, including recorded-tap failure corroboration. #2450 needs to land with this change, or comparison across this boundary must be explicitly refused. Selector responses also drop the disclosure because they only read Android systemSurfaceOnly; include #2451 or narrow the advertised contract explicitly.

Coverage fails on a related import-closure regression: app-lifecycle-facade grows from 120 to 121 modules through the new static import. Fix the import boundary without raising the budget. The iOS failure is an initial fixture-app capture stall, which looks unrelated to the auth route.

Please itemize the +8.1 kB unpacked growth and briefly record the smaller design considered. The reported live sign-in flow is useful evidence, but does not cover the failure paths above.

Presence probe: absence and probe failure are no longer reported as "no
surface". The probe returns present/absent/unknown and the route takes the
runner for anything but a proven absent, so a sheet opened between two captures,
or a probe that cannot answer, can no longer fall through to a bridge capture
that would answer confidently from the occluded app tree. Only a positive
observation is memoized. The probe now matches with pgrep and reads only a
matched pid's environment, which is ~3x cheaper than the previous full
process-environment dump and stops copying every process's environment.

Open guard: the refusal moved to every resolved-host launch and terminate, so
the URL, deep-link and launch-args branches that returned before the old check
can no longer launch the host. Terminating a host is refused too, since that
cancels the presented session just as launching it does.

Comparison: the surface identity now reaches SnapshotState, and tap-failure
corroboration refuses outright when a baseline and a post-action capture
disagree about it, instead of letting app and sheet captures meet in legacy
same-presentation matching. Selector routes disclose an iOS system surface
through the shared disclosure seam rather than reading only the Android field.

The contracts import in the launch path is deferred so the app-lifecycle
facade's eager closure stays flat, and the runner's comment prose is trimmed
because apple/runner ships to npm as uncompiled source.
The probe shelled out with runCmd, so every eligible capture spawned a real
process even in provider-backed tests that stub the Apple tool seam — 17 real
spawns in one scenario file, which is both wasted work and added latency on
timing-sensitive settle paths. It now goes through runAppleToolCommand like the
sibling ps probe, so a stubbed provider answers instead of spawning.
Routing an unprovable probe to the runner is right, but the early return also
skipped runFallback, so the response lost its warning and kept an identity that
could still be compared against a bridge publication. An unknown probe now falls
back through the same disclosed path as a bridge failure, with its own reason.
@thymikee

Copy link
Copy Markdown
Member Author

Addressed at 00e4185. All four gaps were real; two more surfaced while verifying them.

Presence probe. It now returns present/absent/unknown, and the route takes the runner for anything but a proven absent. Only a positive observation is memoized, so a sheet opening between two captures is seen by the very next one, and a probe that cannot answer no longer reads as absence. Tests cover absent→present inside the TTL, unknown→present inside the TTL, and each failure mode (scan failure, unreadable environment, non-"no match" exit). It also stopped dumping every process's environment: pgrep -f first, then a single ps eww on a matched pid only. ~25 ms vs ~170 ms, so re-probing every non-present capture is affordable.

Open guard. Moved to every resolved-host launch and terminate, so the URL, deep-link and launch-args branches that returned before the old check are covered — parameterized tests assert all four plus that no simctl command ever names the host bundle. Terminate is refused too, since that cancels the session just as launching does.

Provenance. #2450 and #2451 are folded in rather than deferred. The surface identity reaches SnapshotState as iosSystemSurfaceBundleId; hasMatchingPresentation refuses outright when baseline and post-action disagree, so app and sheet captures cannot meet in legacy matching; and withSystemSurfaceDisclosure emits the iOS disclosure on the shared selector seam instead of reading only the Android field. Regression tests for both.

Import closure. The contracts import in the launch path is deferred, so app-lifecycle-facade stays at 120. No budget raised.

Two more found while verifying. The probe used runCmd instead of the Apple tool seam, so it spawned real processes even in provider-backed tests (17 in one scenario file); it now goes through runAppleToolCommand like the sibling ps probe. And routing an unprovable probe to the runner had returned early past runFallback, silently dropping the warning and leaving a comparable identity — it now falls back through the disclosed path with its own reason.

iOS smoke. Pre-existing, not this PR. The identical signature (wait_capture_stalled, readableCaptures: 0, one poll consuming the deadline, same step and scenario) occurs on main before the probe existed — runs 34400074702 (main, 09-09, overshoot 7 ms) and 34226333904 (main, 09-08, overshoot 1341 ms) versus 608 ms here. It is already tracked as #2343, whose own measurements account for ~9.2s inside the capture with no probe present, and that job fails on main at ~39% (7/18 recent runs). A 3s-bounded probe cannot produce a 10.6s single-capture stall. Caveat kept honest: the probe does add unconditional latency, so it can nudge a case that already had 7 ms of margin — which is part of why it is now ~7x cheaper.

Size, itemized (measured at the reviewed commit). +4,273 B is Swift: apple/runner/ ships as source into dist/apple/runner/, so RunnerSystemSurfaceHostPolicy.swift (1,448 B post-strip), RunnerTests+CommandExecution.swift (+2,302 B) and RunnerTests+Models.swift (+523 B) all land in the tarball; the #if AGENT_DEVICE_RUNNER_UNIT_TESTS block (1,596 B) was correctly stripped. +1,964 B is TypeScript: the new ios-system-surface.js chunk (808 B, ~570 B of it the two agent-facing strings), the probe (531 B), 476 B of inline hunks, 149 B of cross-chunk imports. Tests, the fixture, the ADR, the subpath export and the layering snapshot add 12.4 kB of source and ship zero bytes; public .d.ts is unchanged. The ~1.9 kB gap to 8.1 kB is diffuse minifier chunk-graph churn, which is why download moved only +1.7 kB.

Smaller design considered. Collapsing the contracts module into an existing one saves only ~200 B, because most of that chunk is the irreducible refusal/disclosure strings — not worth conflating the registry with an unrelated module. The real lever was Swift comment prose, which ships uncompiled: I trimmed the new blocks to pointers at the ADR/TS twin. That also turned up a repo-wide finding — comments are 16.6% of the 446 kB of shipped runner Swift (~74 kB) — filed as #2461 with a PR in flight.

Gates: pnpm check:affected --run green at 00e4185.

@thymikee

Copy link
Copy Markdown
Member Author

Still needs changes at 00e4185:

  • A successful ps read without SIMULATOR_UDID is treated as absence. Missing or ambiguous device scope must remain unknown; otherwise a live sheet can be routed to the occluded app tree. Add a missing-environment regression.
  • Generic post-gesture comparison still uses comparisonKey or the backend, without the new system-surface identity. App and sheet XCTest captures can therefore remain comparable. Carry surface identity through the shared comparison path and cover sheet appearance/dismissal in stabilization, verify and settle.
  • Mutating find drops iosSystemSurfaceBundleId in find-target-capture.ts, so the shared disclosure helper cannot report the sheet. Preserve it and test both found and not-found responses.

The current size report is +10.6 kB unpacked, while the explanation accounts for +8.1 kB. Please reconcile the remaining growth against this head. CI is green, but these code gaps remain.

…cope

A ps read that carries no SIMULATOR_UDID at all was reported as absence, so an
unreadable or truncated environment could route a live sheet to the occluded app
tree. Only a scope naming a different device is a real negative now; a missing
one stays unknown.

The shared post-gesture comparison token used comparisonKey or the backend
alone, so an app capture and a sheet capture — both XCTest — compared equal and
a sheet appearing or dismissing read as a stable surface. The token now carries
the surface, which covers stabilization, verify and settle through the one path
they share.

Mutating find rebuilt its capture without iosSystemSurfaceBundleId, so the
shared disclosure helper could not report the sheet on either outcome. It is
preserved now.

Each fix has a regression that fails without it.
@thymikee

Copy link
Copy Markdown
Member Author

Addressed at 2200150. All three code gaps were real, and each fix has a regression that fails without it (verified by reverting the fix and watching the new test go red).

Probe device scope. A successful ps read carrying no SIMULATOR_UDID at all was reported as absence. Now only a scope naming a different device is a real negative; a missing or unreadable scope stays unknown and takes the runner. Two regressions: an environment with no scope token, and an empty read.

Generic post-gesture comparison. You were right that this was still open — snapshotComparisonKey returned comparisonKey ?? backend, so an app capture and a sheet capture, both XCTest, compared equal on backend alone. The token now carries the surface identity, which covers stabilization, verify and settle through the one path they share. Regressions for both directions: a sheet appearing mid-poll and a sheet dismissing mid-poll, each asserting a re-baseline rather than a verdict.

Mutating find. find-target-capture.ts rebuilt its capture without iosSystemSurfaceBundleId, so the shared helper could not see it. Preserved now, with tests on both the found and the not-found outcome plus a direct helper test.

Size, reconciled against this head. Reproducing CI's own definition locally (tsdown + package-apple-runner-source.mjs + npm pack --json unpackedSize), +10.8 kB is +10,778 B: dist/src/** +7,162 and dist/apple/runner/** +3,616, with every other shipped path at exactly 0.

Of that, +8,908 B is this branch's own code measured against the merge-base — Swift +3,122 (RunnerSystemSurfaceHostPolicy.swift +1,016 new, RunnerTests+CommandExecution.swift +1,798, RunnerTests+Models.swift +308) and dist/src +5,786, the largest items being sdk-selectors.d.ts +1,770, runtime4.js +1,393, the new ios-system-surface.js chunk +1,041, interactor.js +410, sdk-contracts.d.ts +348, app-launch.js +285. The remaining +1,870 B is base drift: main shrank by exactly that between the merge-base and this PR's base sha, so the diff against a smaller base reads larger by the same amount.

The 8.1 → 10.6 → 10.8 walk is the review fixes, per commit: 1d253c4c +3,457 B of dist/src while giving back −1,151 B of Swift comment prose, then −12, +159, +184. 8,141 + 2,637 = 10,778 exactly.

Correction to my earlier comment: the "~1.9 kB of diffuse minifier churn" I claimed does not exist. That figure was a base mismatch — I itemized against the merge-base and compared it to a CI number measured against the PR base sha. Rebuilding the same sha in two independent directories yields byte-identical dist/src totals, so the build is deterministic and the residual after this itemization is 0 B. One genuine item I had also missed: .d.ts keeps doc comments even though the minified .js drops them, so the new contracts module costs +2,118 B of declarations — that, not churn, is where the surprise was.

Gates: pnpm check:affected --run green at 2200150.

@thymikee

Copy link
Copy Markdown
Member Author

The probe and mutating-find fixes are addressed at 2200150, and the size accounting now reconciles. The verify/settle gap remains: the new key only protects deferred post-gesture stabilization. captureVerifyEvidence and settleEvidence still compare node digests, while readSettledOutcome diffs node-only baselines; none compares the captured surface identity. Carry that identity through these actual command paths and test app-to-sheet and sheet-to-app transitions through --verify, --settle, and their combination. The new deferred-stabilization tests do not exercise those routes.

thymikee added a commit that referenced this pull request Sep 10, 2026
An eligible iOS simulator snapshot is served by the host AX bridge
(packages/platform-apple/src/snapshot-route.ts), which never reaches the
runner's prepareActiveCommandContext. The rescue's direct querySelector always
does, so the two requests share that surface policy only when the capture is
runner-routed too - which is the case #2448 forces for the system surface.
Keep the unchanged statement that no surface identity crosses the two requests.
`--verify` compared node digests and `--settle` diffed node-only baselines, so an app
baseline and an in-place system-surface capture (a web sign-in sheet) were treated as one
presentation: a meaningless changed verdict, and a whole-surface replacement presented as an
in-surface diff with refs.

The pre-action baseline now travels with the surface its capture described, from the resolution
and the session frame through to the settled capture, and one module owns the comparison for
both routes. Across a surface change no same-surface claim is made: evidence reports the
transition instead of a digest comparison, the settled diff and its refs are withheld, and both
payloads disclose the transition.
@thymikee

Copy link
Copy Markdown
Member Author

Verify/settle gap addressed at 36fdde3.

Identity now travels with the baseline, not beside it. ResolvedInteractionTarget's pre-action fields are one member type (preActionNodes + preActionSurfaceBundleId), emitted from a single preActionBaselineFields helper in resolution.ts wherever nodes are taken — point evidence baseline, ref resolution (RefResolution carries the surface), selector capture, native-ref preflight, and the find handoff (PreresolvedInteractionTarget.iosSystemSurfaceBundleId, set from find's own target capture). SettleOutcome returns settledCapture: { nodes, surfaceBundleId } instead of bare settledNodes; resolveSettleBaseline keeps the surface on all three baseline sources (authorized ref frame, resolution evidence, session fallback), and the generic scroll/back route passes baselineSurfaceBundleId from the stored snapshot.

One module, src/commands/interaction/runtime/post-action-surface.ts, owns the comparison for both routes, so the refusal cannot hold on one and drop on the other.

What a cross-surface verify/settle now does

  • --verify: no digest comparison is made across the boundary. evidence.changedFromBefore reports the transition itself (the whole observed surface was replaced) and evidence.surfaceChange = { from, to, disclosure } says so — from/to are the host bundle id or app.
  • --settle: no settled diff is attached, so (since diff presence is what issues refs) no refs are issued and no tail either; settle.surfaceChange carries the same triple and hint states the transition plus "take a snapshot to read the current surface". The loop's own verdict (never settled, stalled, sparse, tiny tree) is still appended, not replaced.
  • --settle --verify: the shared final capture feeds settleEvidence, which routes through the same rule, so both payloads report one consistent transition.
  • Disclosure text reuses IOS_SYSTEM_SURFACE_DISCLOSURE when the sheet is now on screen; a new iosSystemSurfaceTransitionDisclosure adds the sheet-has-left sentence, since the standing present-tense one cannot say that.
  • The settled sheet still becomes the stored observation, and the non-hittable "may have had no visible effect" hint is dropped when a surface change is disclosed next to it.

Same-surface behavior is byte-identical: all 55 pre-existing verify/settle tests pass unchanged (two accessor renames in settle-transition-baseline.test.ts for settledCapture.nodes; assertions untouched).

New tests (6, beside the existing verify/settle tests) cover app→sheet and sheet→app through each route: --verify (both directions, in interactions.test.ts), --settle (both directions) and --settle --verify (both directions) in settle.test.ts. Two of them pin the premise that the sheet and app trees digest identically, so the old digest comparison would have reported "nothing changed" across a whole-surface replacement.

Revert-sensitivity — three separate reverts, each confirmed:

  1. comparison ignores surface identity → all 6 new tests fail, 55 existing pass;
  2. diff gating removed (!surfaceChange dropped) → the 4 settle-route tests fail;
  3. surface dropped from the baseline plumbing → exactly the 3 sheet→app tests fail (app→sheet passes by accident when the baseline surface is absent, which is why both directions are covered).

Gates on the pushed commit: pnpm format (oxfmt, clean), pnpm lint, pnpm typecheck, pnpm check:affected --run — all runnable checks passed (full set selected; vitest-related ran 774 files / 5879 tests). No device or simulator use.

@thymikee

thymikee commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

The verify/settle comparison fix looks correct at 36fdde3, including both transition directions and the combined press path. Two validation gaps remain: add an app-to-sheet and sheet-to-app regression through generic scroll/back --settle, where the baseline is passed separately, and exercise the new verify/settle transitions on the live sign-in flow. The earlier live sign-in evidence does not cover these new outputs. The generic-route test should fail if its surface identity is dropped and assert that no cross-surface diff or refs are returned.

Please also update the size breakdown for the current +14.8 kB report; the previous accounting covers +10.8 kB.

Coverage has now failed on a related test-size guard: settle.test.ts grew from 2,359 to 2,528 lines. Split the new surface tests along their owning module, preserving command-path coverage, rather than raising the limit. Native smoke checks are still running. The readiness hold is for the missing evidence, not CI alone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant