Host runner disconnects for CI/CD testing stages when processing through merge queue

[wavefunction91]

For PRs (even merged with main) that pass local CI/CD, merge queue fails with the following disconnect

For example #408, but this problem occurs consistently starting on 3/26/2026.

[wavefunction91]

While not definitive, looking at the Azure logs and instances on different pipelines running on the same runners, this seems to be a disk issue

Rationale for this being that PR CI/CD does a thin clone, while the merge queue will pull the full history and do squash/rebase. As the regular pipeline already only leaves ~5% disk left on the runner, it's quite possible that the full clone pushes things over the edge.

[lorisercole]

While not definitive, looking at the Azure logs and instances on different pipelines running on the same runners, this seems to be a disk issue

Rationale for this being that PR CI/CD does a thin clone, while the merge queue will pull the full history and do squash/rebase. As the regular pipeline already only leaves ~5% disk left on the runner, it's quite possible that the full clone pushes things over the edge.

This actually refers to the ADO wheel build pipelines: #421.

It turns out the errors we are seeing in the GitHub pipelines, like

The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

or

##[error]The runner has received a shutdown signal...
##[error]Process completed with exit code 143.

are due to an infrastructure problem when using 1ES hosted runners, for which we filed a ticket.