Symptom
iOS live lanes (Run fixture-backed iOS simulator E2E smoke, Run iOS Settings replay smoke test) fail a wait that follows open with wait timed out for …, on unrelated branches and on main. Seen 2026-09-10 on main f4c8f3ddda (16:44), on #2473's branch (16:15, passed on retry), on #2471 (17:00), and earlier on 2026-09-06/07 on at least four unrelated branches.
What the request logs show
From the artifacts of runs 34503978665 and 34505591306 (session request ndjson):
open finishes ok in ~4 s.
- The
wait first tries the AX bridge: ios.snapshot-source.prepare / .acquire end with request canceled at ~5.2 s (the snapshot-source default maxDurationMs is 5 s and its signal aborts the in-flight probe or compile, so the error kind is cancelled).
- The runner fallback then logs
ios_runner_session_reuse → ios_runner_connect ok in 7–59 ms → ios_runner_readiness_preflight with timeoutMs: 1000 → a second ios_runner_connect that stalls 4.5–5.1 s → the wait's own deadline cancels it, captures: 1, readableCaptures: 0, and the wait fails at 10.4 s.
- A green run without any of the suspected changes shows the same second connect at 11, 12, 18, 19, 75, 203, 1073 and 2323 ms. Same shape, same budgets (preflight 1000 ms, connect ~45 000 ms, command ~44 9xx ms), just inside the margin. A later plain
snapshot in the failing session succeeds in ~6 s: the runner is alive, it was busy.
Reading: the 1 s readiness preflight abandons a runner command that the runner keeps executing for ~5 s; the next connect waits behind it; a 10 s wait that already spent 5 s on the bridge has nothing left. Cold hosts make the runner's first commands slow enough to cross the line.
Not the cause
#2473 was suspected and reverted in #2474; the revert was withdrawn after a local reproduction on a booted simulator passed 4/4 on both f4c8f3ddda and its parent with identical request logs, and after the CI budgets were found byte-identical between red and green runs. Two of the correlated failures were a Swift-only XCTest step on a different Xcode image (ALERT_DEADLINE_EXCEEDED), unrelated.
Required behavior
- A readiness preflight that times out must not leave the runner executing an abandoned command that blocks the next connect, or the next connect must not wait behind it. Options: make the preflight command idempotent and cheap enough that 1 s is honest on a cold host, cancel it on the runner side when the client abandons it, or size the preflight from the request's remaining budget instead of a fixed 1 s.
- A
wait whose snapshot-source attempt consumed ~5 s must hand the runner fallback a real remainder, and the fallback must not spend it on a connect that cannot answer.
- Evidence to close: the fixture smoke and the settings replay green on a cold runner across several consecutive main runs, and a unit test that plants a 5 s busy runner behind a 1 s preflight and shows the wait still answers within its budget.
Non-goals
Widening the wait's budget in the smoke tests; retrying the smoke harder.
Symptom
iOS live lanes (
Run fixture-backed iOS simulator E2E smoke,Run iOS Settings replay smoke test) fail awaitthat followsopenwithwait timed out for …, on unrelated branches and on main. Seen 2026-09-10 on mainf4c8f3ddda(16:44), on #2473's branch (16:15, passed on retry), on #2471 (17:00), and earlier on 2026-09-06/07 on at least four unrelated branches.What the request logs show
From the artifacts of runs 34503978665 and 34505591306 (session request ndjson):
openfinishes ok in ~4 s.waitfirst tries the AX bridge:ios.snapshot-source.prepare/.acquireend withrequest canceledat ~5.2 s (the snapshot-source defaultmaxDurationMsis 5 s and its signal aborts the in-flight probe or compile, so the error kind iscancelled).ios_runner_session_reuse→ios_runner_connectok in 7–59 ms →ios_runner_readiness_preflightwithtimeoutMs: 1000→ a secondios_runner_connectthat stalls 4.5–5.1 s → the wait's own deadline cancels it,captures: 1, readableCaptures: 0, and the wait fails at 10.4 s.snapshotin the failing session succeeds in ~6 s: the runner is alive, it was busy.Reading: the 1 s readiness preflight abandons a runner command that the runner keeps executing for ~5 s; the next connect waits behind it; a 10 s
waitthat already spent 5 s on the bridge has nothing left. Cold hosts make the runner's first commands slow enough to cross the line.Not the cause
#2473 was suspected and reverted in #2474; the revert was withdrawn after a local reproduction on a booted simulator passed 4/4 on both
f4c8f3dddaand its parent with identical request logs, and after the CI budgets were found byte-identical between red and green runs. Two of the correlated failures were a Swift-only XCTest step on a different Xcode image (ALERT_DEADLINE_EXCEEDED), unrelated.Required behavior
waitwhose snapshot-source attempt consumed ~5 s must hand the runner fallback a real remainder, and the fallback must not spend it on a connect that cannot answer.Non-goals
Widening the wait's budget in the smoke tests; retrying the smoke harder.