There are some bugs in the ubuntu-24.04-arm image- some environment variables are incorrect:
jupyterhub/zero-to-jupyterhub-k8s#3605
I don't know if this is affecting your builds.

Beautiful!!! ❤️ 🎉

A job failed in this PR, but not in the main branch - a difference between these jobs is the test runner's python version, which is 3.10 (main branch) vs 3.13 (in this PR) ---

Then in the x86_64 test from main branch and this PR, the Python test runner version is 3.12:

Maybe the Python version in the aarch image is floating around a bit? Perhaps because the self-hosted aarch runners have a 3.x mapped to 3.10, while the new aarch runners have a 3.x mapped to 3.13, and the x86 runners have 3.x mapped to 3.12?

I figure the fix could be to pin to 3.12

I pushed a commit to see if it helps to pin to 3.12!

Edit: it seems so, or it was a coincidence - now it works at least!

Thanks @consideRatio!
Let's try to restart it a few times to see if it works.

zstd: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? - I don't think Docker is stable enough in these runners (and I suspect that might have been the problem with the previous failure as well).

Oh, hmmm, i wonder if it could be a timing thing - were the CI scripts started faster than docker daemon in arm64 compared to in x64.

These things are messy to debug :/

Oh, hmmm, i wonder if it could be a timing thing - were the CI scripts started faster than docker daemon in arm64 compared to in x64.

These things are messy to debug :/

I think we have quite a nice example of this happening!

This only runs docker build and nothing else.
https://github.com/jupyter/docker-stacks/actions/runs/12999905803/job/36256210849?pr=2202

I added a sleep 10 - let's see if this eliminates the problem

Wow, it seems that even @actions/checkout can fail (possibly for the same reason): https://github.com/jupyter/docker-stacks/actions/runs/13000122202/job/36256874950?pr=2202#step:3:20

And we can't have this workaround in create-dev-env (because we need to checkout first)

It seems that GitHub-hosted aarch64 runners aren't currently stable enough. So, let's wait for a while, I hope it will be better.

The relevant discussions: https://github.com/orgs/community/discussions/148648