Skip to content

samsung device farm stopped assigning devices, test-samsung-models-linux times out on every trunk run #24869

Open

Description

@shoumikhin

since about 14:00 utc today (nov 10), trunk / test-samsung-models-linux never gets a device. devicefarm-cli -Q prints "Queue registration successful", then every devicefarm-cli -C poll returns "Device not assigned. waiting for assignment..." until the job hits its 360 minute cap and gets cancelled.

the last two jobs that had a device released them cleanly ("Device disconnected successfully" at 14:06 and 14:59 utc). jobs that were already waiting in the queue at that point never got assigned. 5 of 5 finished trunk runs on main since then timed out this way (the latest at 21:46 utc), and about 15 more are waiting now. trunk is required for viable/strict, so it cannot move.

could someone on the samsung side check the device farm queue?

also, acquire_device in .ci/scripts/setup-samsung-linux-deps.sh polls with no limit, so a stuck queue holds a runner for 6 hours per run. a cap of 30 to 60 minutes would fail faster and free the runner.

cc @SS-JIA @Jiseong-oh @digantdesai @kimishpatel

Activity

  1. added
    module: ciIssues related to continuous integration
    partner: samsungFor backend delegation, kernels, demo, etc. from the 3rd-party partner, Samsung
    on Nov 10, 2026
  2. pk954061458 commented on Nov 11, 2026

    @pk954061458

    I've tested a local patch that caps acquire_device polling at 30 minutes and puts a deadline on each devicefarm-cli -C call. It covers waiting after -Q succeeds, not the farm outage.

    18 tests pass with a mock CLI, including hung calls and processes that ignore TERM. No Samsung hardware or full CI validation yet. The patch was prepared with OpenAI Codex.

    Would you accept a PR for this? Should a timeout fail the job or warn and skip, and would you prefer 30 or 60 minutes? Is there a supported way to cancel a queued request on timeout?

  3. Jiseong-oh commented on Nov 11, 2026

    @Jiseong-oh
    Collaborator

    Hi, thanks for the proposal. We are internally reviewing this issue.

    Currently, we are running tests on a limited set of devices. Since most of the time is spent on compilation, the total duration depends heavily on how long compilation takes. If we can separate the compilation step, collect the resulting .pte files, and then run them via the devicefarm CLI for files from other workflows, the execution time will be predictable. In that case, we should be able to define a specific timeout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    module: ciIssues related to continuous integrationmodule: samsungpartner: samsungFor backend delegation, kernels, demo, etc. from the 3rd-party partner, Samsung

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions