jaccl RDMA crashes on Mac Studio M3 Ultra (errno=2, errno=60, errno=22) — recurring in production

[Sofille65]

Description

We run a 3-node EXO cluster on Mac Studio M3 Ultra connected via Thunderbolt 5 (RDMA). We experience recurring jaccl RDMA crashes that require a full reboot of all nodes to recover. This happens multiple times per day under normal inference workloads.

Errors observed

Three distinct errors, all from jaccl:

1. errno=2 — Connection lost mid-inference

RuntimeError: [jaccl] Recv failed with errno=2

2. errno=60 — Connection timeout

RuntimeError: [jaccl] Recv failed with errno=60

3. errno=22 — Queue pair negotiation failure (most frequent)

ValueError: [jaccl] Changing queue pair to RTR failed with errno 22

The errno=22 error appears simultaneously on all nodes and seems to indicate a hardware-level RDMA state corruption — the queue pair cannot transition to RTR (Ready To Receive) state.

Trigger conditions

• Happens after extended use (multiple model loads/unloads per day)
• Can happen mid-inference or when loading a new model
• Once triggered, all nodes crash in a loop — the cluster never recovers without reboot
• Purging ~/.exo/event_log/ alone is not sufficient — full reboot required

Recovery

# On all nodes:
rm -rf ~/.exo/event_log/
osascript -e 'tell app "System Events" to restart'

Is there a way to recover without rebooting?

We're looking for a way to reset the jaccl/RDMA state programmatically without requiring a full OS reboot. Any mlx.core.distributed or jaccl-level teardown/reinit API that could help?

Environment

• EXO.app v0.3.68 (commit 5327bddd)
• macOS 26.3.1 (build 25D2128)
• Hardware: 3× Mac Studio M3 Ultra (1× 512GB + 2× 256GB RAM)
• Interconnect: Thunderbolt 5 mesh (RDMA via jaccl)
• Load: ~10 model load/unload cycles per day, continuous inference

[Sofille65]

Additional observation: crash happens at idle, no inference running

After further investigation, we confirmed that the RDMA crash can happen spontaneously without any active inference or model load.

Timeline from today:

• 11:42 → last activity (background HF polling, no inference)
• 11:42 to 12:05 → 23 minutes of idle — no TextGeneration commands, no model loads
• 12:05:43 → jaccl ConnectToGroup attempt triggered
• 12:05:44 → immediate crash: errno=2 on ultra-512, errno=22 on both 256GB nodes simultaneously

This suggests the RDMA queue pair state degrades over time without traffic, and the next connection attempt fails immediately. The TB5/RDMA link may require periodic keepalive traffic to maintain queue pair state.

Is there a jaccl keepalive or heartbeat mechanism? Or a way to detect stale queue pairs before attempting ConnectToGroup?

[lucasajackson]

Additional crash path on the same hardware class: SIGSEGV in tbt_post_recv (not errno=N)

Hitting this on a 4× Mac Studio M3 Ultra / Thunderbolt / RDMA cluster on EXO.app (Resources/exo binary built against macOS 26.3 / 25D125). Our symptom is a hard SIGSEGV in the RDMA receive path rather than the errno=2/22/60 RuntimeError pattern in the main body of this issue — likely a related / adjacent failure mode rather than the exact same one, but same subsystem and same hardware class.

Symptom at the cluster level

One node (Red) crashed its multiprocessing-fork worker at 2026-04-21 09:44:48 -0400. The main exo process survived, so:

• Red's main proc keeps running and continues to answer /state locally, but its topology shows only itself (1 node).
• The other 3 nodes (Yellow / Purple / Green) form a 3-node topology that does not include Red.
• masterId is none everywhere. No instances loaded.
• Red is fully reachable on L3 — ping 192.168.5.13 works from every peer, Thunderbolt interfaces all have 169.254.x addresses, EXO network location active. The partition is at the Exo / JACCL layer, not the OS network layer.
• Red's own nodeIdentities cache still has entries for all three peers (plus a stale Yellow libp2p ID from before Yellow last restarted), so Red was previously in the cluster and was silently ejected after the worker crash — the main proc never re-enrolled.

Crash signature (Red, 09:44:48)

Exception: EXC_BAD_ACCESS (SIGSEGV), KERN_INVALID_ADDRESS at 0x111368008

libthunderboltrdma.dylib    tbt_post_recv
libthunderboltrdma.dylib    tbt_post_recv
libjaccl.dylib              jaccl::Connection::post_recv(jaccl::SharedBuffer const&, unsigned long long)
libjaccl.dylib              jaccl::RingImpl::all_reduce<2, jaccl::bfloat16_t, jaccl::SumOp<jaccl::bfloat16_t>>(
                              jaccl::bfloat16_t const*, jaccl::bfloat16_t*, long long, int, ...)
libmlx.dylib                mlx::core::distributed::jaccl::(anonymous namespace)::JACCLGroup::all_sum(...)
libmlx.dylib                mlx::core::scheduler::StreamThread::thread_fn()
libsystem_pthread.dylib     _pthread_start / thread_start

Identical stack on Purple at 09:44:48 the same morning, and again on Purple + Green at 11:59:41/43. Four separate fork-child SEGVs with the same frames.

Why this matters

• In the errno=2/22/60 path documented in this issue, the failure surfaces as a catchable RuntimeError / ValueError and Exo can at least log it before aborting. In the SEGV path, the worker dies without a log entry and the main process is left in a zombie state: /state still serves, /v1/models still responds, but the node has no RDMA transport and gets orphaned from the cluster. Users see "the cluster quietly went from 4 to 3" with no obvious error.
• Recovery is the same — full OS reboot is the only reliable path — but the detection story is worse because nothing in Exo's own logs says "I crashed." You only notice it by diffing topology.nodes across /state endpoints and seeing that one node lists just itself.

Could be useful as defensive fixes in Exo itself

• Wrap jaccl::Connection::post_recv in a guard that rechecks the SharedBuffer and queue-pair state before calling ibv_post_recv, so a corrupted QP raises an errno (falling into the existing jaccl RDMA crashes on Mac Studio M3 Ultra (errno=2, errno=60, errno=22) — recurring in production #1847 code path) rather than hitting an invalid pointer. That would unify this failure mode with the ones you already track.
• When the spawned --multiprocessing-fork worker exits for any reason, have the main proc either crash loudly or eject itself from its own topology. Right now the main proc survives, keeps heartbeating at libp2p, but has no working RDMA — so upstream peers assume it's fine and it silently drops out of theirs.
• A /health endpoint that distinguishes "libp2p up" from "JACCL worker alive" would let operators detect this without parsing diagnostic reports.

Environment

• EXO.app (/Applications/EXO.app/Contents/Resources/exo/exo), codeSigningTeamID AF25623428
• 4× Mac Studio M3 Ultra, 256GB each, macOS 26.3 (25D125)
• Thunderbolt 5 mesh, network location exo, Thunderbolt Bridge disabled, all EXO Thunderbolt 1–6 services on DHCP link-local
• No model instance loaded at time of failures on any node (Red had a model loaded earlier, got crashed out of all_reduce, and never re-joined)

Happy to provide full .ips files if useful — they contain the instructionByteStream and the fault address 0x111368008 which might help narrow down which pointer in tbt_post_recv is being dereferenced.

[guruswami-ai]

Data point: same hardware class, no longer crashing — and a patch stack that holds up

Sharing a positive update from a similar hardware setup, in case it's useful to others hitting this issue.

Hardware

• 5× Mac Studio M3 Ultra (1× 512GB + 4× 512GB), Mac mini M4 Pro orchestrator
• Thunderbolt 5 full mesh (RDMA via jaccl)
• macOS 15.4
• Daily distributed inference workload (Kimi K2.6, ~600 GB MoE model; PP2/PP3/TP4 topologies)

What used to happen (matches this issue exactly)

• errno=22 / errno=60 on Changing queue pair to RTR failed — most often after a daemon kill/restart, even on otherwise-healthy ports
• errno=2 Recv failed mid-inference under load
• Silent SIGSEGV in the tbt_post_recv path (matches @lucasajackson's observation above) — one rank dies, main proc survives but topology falls apart
• Recovery required full reboot of all participating nodes; nothing released the protection-domain state without a power cycle
• Almost every failed inference run needed cluster-wide reboot — operationally unusable

What's changed for us (testing today, 2026-05-18 AEST)

Today we ran 5+ deliberate PP2 daemon crashes (Metal kernel timeouts on long-context prefill), 2 PP3 daemon crashes, and 1 JACCL handshake hang during diagnostics. Zero node reboots required between failures. JACCL/RDMA state survived every crash. Mesh stayed healthy throughout. This is night-and-day from the K2.5 / pre-patch era.

Patch stack that gets us here

Our mlx build is at commit d846fe21, which is eaa0144b + several cherry-picks. Relevant for this issue, in order of how much they help with the symptoms here:

Commit	Upstream	What it fixes	State
5efbedd4	#3451	Race on local_staging in MeshImpl::all_reduce	Merged 2026-05-11
30aee33a	#3459	Add barrier to JACCL	Merged 2026-04-28
eaa0144b	#3152	Silent data corruption from unchecked RDMA work completion status (wc_status)	Open (maintainer benchmarked ~10% latency on small all_reduce as the holdup)
fe88a791	#3142 (WONTFIX upstream)	MTL::ResourceHazardTrackingModeUntracked → ModeDefault workaround	Local patch — maintainer said no public CPU/GPU coherence API
d846fe21	rltakashige/mlx-jaccl-fix-small-recv #1 (vskiwi)	__dsb(0xF) read-side barrier in fence wait spin loop	Open against the JACCL fork

The #3451 race fix is the single biggest contributor based on our observations — it eliminates the RTR errno 22/60 failures that happen after a clean-killed daemon leaves QPs in an in-flight state. Before this patch, almost every test cycle needed a full reboot. After it (combined with the others), we can restart daemons freely.

What still fails — but predictably and recoverably

• Metal 60s per-kernel timeout still binds at long contexts. PP2 caps single-stream around ~7-8K tokens (Kimi K2.6 q4); PP3 caps lower (~5-6K tokens). The kernel exceeds 60s on the last prefill chunk due to attention scaling × cross-rank coordination overhead. This is a Metal limit, not a JACCL bug — the daemon crashes cleanly and the cluster stays up.
• Single-node mlx_lm.server handles 24K+ context on the same hardware without this issue, because chunked prefill keeps individual kernels under 60s when there's no cross-rank handoff per layer.

Concrete numbers from today

PP2 (Kimi K2.6 q4-affine, sva+man M3 Ultras):

• Single-stream decode: 0.152 TPS
• 4 concurrent streams: 0.567 TPS aggregate (3.73x, 93% scaling efficiency) — mlx-lm's --prompt-concurrency correctly schedules across PP ranks, the pipeline bubble fills with batching
• Per-rank RSS: 299.6 GB rank 0 / 289.6 GB rank 1 (≈ 590 GB / 2 — half the model on each, as expected for PP)

PP3 (sva+man+ana):

• Single-stream: 0.189 TPS
• 4 streams: 0.498 TPS aggregate (2.64x scaling — weaker than PP2)
• Per-rank RSS: ~190-200 GB (≈ ⅓ of model on each)

RDMA broadcast (mlx_lm.share) anahata → vishuddha: 597 GB transferred in 1:58 at 5.4 GB/s sustained. Reliable.

Practical envelope (Apple Silicon distributed inference as of today)

• Single-user, long-context (interactive) → single-node mlx_lm.server, up to whatever fits in one M3 Ultra's RAM. Reliable.
• High-concurrency short-context serving (RAG fan-out, multi-agent, image batching) → PP2 with --prompt-concurrency. Works, scales well.
• Model too large for one node → PP4/PP5 to fit the weights, but single-stream context will be small (extrapolating: PP4 ~4K, PP5 ~3K).
• Anything beyond that envelope → cloud / NVIDIA. Apple now has predictable limits rather than mysterious crashes.

Open published data

The chakra simulator at https://chakra.guruswami.ai was built on K2.5-era benchmarks where most of these failures DID require reboot. We'll be updating it with this new data. Our methodology + raw numbers are in https://github.com/guruswami-ai/mlx-benchmarks.

Asks for upstream

1. Land #3152 — the 10% small-all_reduce latency cost is real but operationally, silent data corruption is a worse trade. Several of us are running it cherry-picked in production already.
2. A way to surface stderr from non-rank-0 in mlx.launch — when a rank N>0 crashes, the only signal is "rank N exited with code 255." The actual exception is on the remote node's stderr but mlx.launch's ssh wrapping eats it. This made the bugs in this thread much harder to diagnose.
3. Document the Metal 60s kernel timeout interaction with PP topologies — the per-rank-fewer-layers benefit is wiped out by cross-rank coordination per kernel. Knowing this in the docs would save a lot of folks the binary search we did today.

Happy to share more detail / specific timing numbers if it helps.

[zhao18235158400]

Hi @Sofille65 — Daniel here, doing user research for a team building a local AI system that can scale to 512GB memory. Your 3-node M3 Ultra cluster report stood out because this is clearly continuous production inference, not a one-off benchmark: about ten model load/unload cycles per day, with RDMA failures sometimes forcing all nodes to reboot.

Would you be open to a 25–30 minute video call about why you chose a 512GB + 2×256GB cluster, what workloads justify that capacity, and where distributed reliability now costs you the most time? This is product research, not a sales pitch. If a call is inconvenient, I can send a few focused questions here instead.

— Daniel

[Sofille65]

@zhao18235158400 Hi Daniel, yes, no problem. I have extended a bit 512 + 256 + 256 + 256 + 256 ; all M3 Ultra.
I'm running GLM 5.2 Q8, Ring 2.6, ...

[zhao18235158400]

Great, thanks. What time zone are you in, and what's the easiest way to reach you? I can send a few times after that.

— Daniel

[Sofille65]

I'm UTC+1 (Paris time)
send me your slots at sophie@themonoclebear.com