Skip to content

ios(runner): a 1 s readiness preflight abandons a busy runner command and the next connect stalls behind it, failing waits on cold hosts #2475

Description

@thymikee

Symptom

iOS live lanes (Run fixture-backed iOS simulator E2E smoke, Run iOS Settings replay smoke test) fail a wait that follows open with wait timed out for …, on unrelated branches and on main. Seen 2026-09-10 on main f4c8f3ddda (16:44), on #2473's branch (16:15, passed on retry), on #2471 (17:00), and earlier on 2026-09-06/07 on at least four unrelated branches.

What the request logs show

From the artifacts of runs 34503978665 and 34505591306 (session request ndjson):

  • open finishes ok in ~4 s.
  • The wait first tries the AX bridge: ios.snapshot-source.prepare / .acquire end with request canceled at ~5.2 s (the snapshot-source default maxDurationMs is 5 s and its signal aborts the in-flight probe or compile, so the error kind is cancelled).
  • The runner fallback then logs ios_runner_session_reuseios_runner_connect ok in 7–59 ms → ios_runner_readiness_preflight with timeoutMs: 1000 → a second ios_runner_connect that stalls 4.5–5.1 s → the wait's own deadline cancels it, captures: 1, readableCaptures: 0, and the wait fails at 10.4 s.
  • A green run without any of the suspected changes shows the same second connect at 11, 12, 18, 19, 75, 203, 1073 and 2323 ms. Same shape, same budgets (preflight 1000 ms, connect ~45 000 ms, command ~44 9xx ms), just inside the margin. A later plain snapshot in the failing session succeeds in ~6 s: the runner is alive, it was busy.

Reading: the 1 s readiness preflight abandons a runner command that the runner keeps executing for ~5 s; the next connect waits behind it; a 10 s wait that already spent 5 s on the bridge has nothing left. Cold hosts make the runner's first commands slow enough to cross the line.

Not the cause

#2473 was suspected and reverted in #2474; the revert was withdrawn after a local reproduction on a booted simulator passed 4/4 on both f4c8f3ddda and its parent with identical request logs, and after the CI budgets were found byte-identical between red and green runs. Two of the correlated failures were a Swift-only XCTest step on a different Xcode image (ALERT_DEADLINE_EXCEEDED), unrelated.

Required behavior

  • A readiness preflight that times out must not leave the runner executing an abandoned command that blocks the next connect, or the next connect must not wait behind it. Options: make the preflight command idempotent and cheap enough that 1 s is honest on a cold host, cancel it on the runner side when the client abandons it, or size the preflight from the request's remaining budget instead of a fixed 1 s.
  • A wait whose snapshot-source attempt consumed ~5 s must hand the runner fallback a real remainder, and the fallback must not spend it on a connect that cannot answer.
  • Evidence to close: the fixture smoke and the settings replay green on a cold runner across several consecutive main runs, and a unit test that plants a 5 s busy runner behind a 1 s preflight and shows the wait still answers within its budget.

Non-goals

Widening the wait's budget in the smoke tests; retrying the smoke harder.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions