Guide: RDMA file transfer over Thunderbolt 5 with JACCL (3.5+ GB/s)

[guruswami-ai]

RDMA File Transfer over Thunderbolt 5 with JACCL

We built a high-speed file transfer tool using MLX's distributed primitives over RDMA/Thunderbolt 5, achieving 3.5–3.8 GB/s sustained throughput — roughly 23× faster than rsync over the same link. Sharing the working solution and the critical workarounds we discovered, since there's no existing documentation for this use case.

Tested on Mac Studio M3 Ultra (macOS 26.3.1) with MLX 0.30+ and JACCL backend.

The Problem

JACCL provides RDMA over Thunderbolt 5, but mx.distributed.send/recv is designed for tensor operations during training and inference, not file I/O. Using it naively for file transfer hits several issues that crash or hang the process. After extensive debugging, we found three critical workarounds and one macOS-level fix that are required.

Results

[RDMA-DIR][rank=0] DONE: 16 files, 49.63 GB in 13.0s (3.80 GB/s)

Method	Speed	235 GB Transfer
RDMA (this approach)	3.5–3.8 GB/s	~67 seconds
rsync over 10GbE	~100–200 MB/s	~30–40 minutes
rsync over TB5 IP	~300–500 MB/s	~10–15 minutes

We distributed 705 GB of quantized LLM weights across 3 nodes in ~3.5 minutes using sequential RDMA transfers between directly-connected pairs.

---

Critical Workaround 1: Use stream=mx.cpu for All Send/Recv

Symptom: Receiver crashes with [METAL] Command buffer execution failed: Caused GPU Timeout Error. Sender reports high throughput but receiver writes zero files.

Root cause: By default, mx.distributed.send/recv are scheduled on the GPU stream. Metal has an unconfigurable command buffer timeout (~5 seconds). RDMA recv operations that take longer than this trigger the timeout and kill the process. The sender doesn't crash because send operations complete quickly once data is in the RDMA buffer, but the receiver must wait for data to arrive.

Fix: Pass stream=mx.cpu on every send() and recv() call:

s = mx.cpu

# Sender
mx.eval(mx.distributed.send(data, peer, group=group, stream=s))

# Receiver
result = mx.distributed.recv(shape, dtype, peer, group=group, stream=s)
mx.eval(result)

This bypasses Metal entirely. The RDMA transfer still happens at full hardware speed — the CPU stream just means MLX doesn't create a Metal command buffer for the operation.

What doesn't work:

• Smaller chunks (16 MB, 4 MB) — timeout still triggers
• mx.eval() after every recv — can deadlock with the sender
• MLX_METAL_FAST_SYNCH=0 — no effect
• Converting to numpy immediately — crash happens before conversion

Related: #3142

Critical Workaround 2: Init Barrier After distributed.init()

Symptom: First send/recv after init returns garbage data or wrong shapes. Subsequent operations may hang or produce corrupted results.

Root cause: JACCL needs all ranks to synchronize after initialization before point-to-point operations work reliably. Without a barrier, one rank may start sending before the other has finished setting up its RDMA resources.

Fix: Run an all_sum barrier immediately after init, before any send/recv:

world = mx.distributed.init()

# REQUIRED: init barrier. Must use mx.ones(10), NOT mx.ones(1).
barrier = mx.distributed.all_sum(mx.ones(10), group=world, stream=mx.cpu)
mx.eval(barrier)

Important: Use mx.ones(10), not mx.ones(1). The single-element variant doesn't reliably trigger the full synchronization path in JACCL.

Related: #3149

Critical Workaround 3: Single JACCL Session (PD Exhaustion)

Symptom: After ~60 successful file transfers, JACCL fails with Couldn't allocate protection domain.

Root cause: Each mx.distributed.init() / teardown cycle allocates and (incompletely) releases RDMA protection domains. The kernel has a hard limit. An approach that launches a separate JACCL session per file hits this limit quickly.

Fix: Transfer all files in a single JACCL session. One init, all transfers, one teardown.

Recovery: If you hit PD exhaustion, the only fix is to reboot the affected node. There is no way to release protection domains without a reboot.

macOS Thunderbolt Bridge (Required Fix)

macOS creates a "Thunderbolt Bridge" network service that absorbs all TB ports into a single bridge0 interface. This completely breaks RDMA — JACCL cannot address individual Thunderbolt ports through a bridge.

Symptom: Every RDMA device combination fails with RTR errno 22 (EINVAL) during queue pair setup.

Critical: macOS re-enables Thunderbolt Bridge on every reboot and OS update. You must run the fix after every boot.

#!/bin/bash
# Fix Thunderbolt Bridge for RDMA — create individual TB port services.
# Run as root: sudo bash fix-tb-bridge.sh
set -euo pipefail

PREFS="/Library/Preferences/SystemConfiguration/preferences.plist"

# 1. Destroy bridge0
if ifconfig bridge0 &>/dev/null; then
  ifconfig bridge0 | awk '/member/ {print $2}' | while read m; do
    ifconfig bridge0 deletem "$m" 2>/dev/null || true
  done
  ifconfig bridge0 destroy 2>/dev/null || true
fi

# 2. Remove bridge from system preferences
/usr/libexec/PlistBuddy \
    -c "Delete :VirtualNetworkInterfaces:Bridge:bridge0" "$PREFS" 2>/dev/null || true

# 3. Create individual network services per Thunderbolt port
networksetup -listallhardwareports \
    | awk -F': ' '/Hardware Port: Thunderbolt [0-9]/ {print $2}' \
    | while read port; do
  svc="RDMA $port"
  if ! networksetup -listallnetworkservices | grep -q "$svc"; then
    networksetup -createnetworkservice "$svc" "$port" 2>/dev/null || true
  fi
  networksetup -setv6automatic "$svc" 2>/dev/null || true
done

# 4. Disable the bridge service
networksetup -setnetworkserviceenabled "Thunderbolt Bridge" off 2>/dev/null || true

# 5. Verify
ibv_devinfo 2>/dev/null | grep -E 'hca_id|state'

After running this, ibv_devinfo should show individual RDMA devices with PORT_ACTIVE state for each connected Thunderbolt cable.

---

Complete Transfer Script

A working script that transfers an entire directory in a single JACCL session with all workarounds applied:

"""
Single-session RDMA directory transfer via JACCL.

Uses CPU stream for all send/recv to avoid Metal GPU timeouts (#3142).
Uses all_sum barrier after init to avoid shape corruption (#3149).

Usage:
  mlx.launch --hostfile hostfile.json --backend jaccl -- \
    rdma_dir_transfer.py /source/dir /dest/dir
"""

import gc
import os
import pathlib
import struct
import sys
import time

import mlx.core as mx

CHUNK_SIZE = 64 * 1024 * 1024  # 64 MB


def log(msg):
    r = mx.distributed.init().rank()
    print(f"[RDMA-DIR][rank={r}] {msg}", flush=True)


def send_directory(src_dir, peer, group):
    files = sorted(f for f in src_dir.iterdir() if f.is_file())

    s = mx.cpu
    cnt = mx.distributed.send(mx.array([len(files)], dtype=mx.int64), peer, group=group, stream=s)
    mx.eval(cnt)

    total = 0
    t0 = time.time()

    for i, fp in enumerate(files):
        sz = fp.stat().st_size
        nm = fp.name.encode()

        hdr = mx.array(list(struct.pack("<QQ", len(nm), sz)), dtype=mx.uint8)
        mx.eval(mx.distributed.send(hdr, peer, group=group, stream=s))
        mx.eval(mx.distributed.send(mx.array(list(nm), dtype=mx.uint8), peer, group=group, stream=s))

        if sz > 0:
            with fp.open("rb") as f:
                sent = 0
                while sent < sz:
                    chunk = f.read(CHUNK_SIZE)
                    if not chunk:
                        break
                    arr = mx.array(memoryview(chunk), dtype=mx.uint8)
                    mx.eval(mx.distributed.send(arr, peer, group=group, stream=s))
                    sent += len(chunk)

        total += sz
        el = time.time() - t0
        spd = total / el / 1e9 if el > 0 else 0
        log(f"  [{i+1}/{len(files)}] {fp.name} ({sz/1e6:.1f} MB) {spd:.2f} GB/s")

    el = time.time() - t0
    spd = total / el / 1e9 if el > 0 else 0
    log(f"DONE: {len(files)} files, {total/1e9:.2f} GB in {el:.1f}s ({spd:.2f} GB/s)")


def recv_directory(dst_dir, peer, group):
    dst_dir.mkdir(parents=True, exist_ok=True)
    s = mx.cpu

    cnt = mx.distributed.recv((1,), mx.int64, peer, group=group, stream=s)
    mx.eval(cnt)
    file_count = int(cnt[0].item())
    log(f"Expecting {file_count} files")

    total = 0
    t0 = time.time()

    for i in range(file_count):
        hdr = mx.distributed.recv((16,), mx.uint8, peer, group=group, stream=s)
        mx.eval(hdr)
        nlen, sz = struct.unpack("<QQ", bytes(memoryview(hdr)))

        nm = mx.distributed.recv((nlen,), mx.uint8, peer, group=group, stream=s)
        mx.eval(nm)
        name = bytes(memoryview(nm)).decode()

        path = dst_dir / name

        if sz > 0:
            fd = os.open(str(path), os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o644)
            try:
                received = 0
                while received < sz:
                    csz = min(CHUNK_SIZE, sz - received)
                    arr = mx.distributed.recv((csz,), mx.uint8, peer, group=group, stream=s)
                    mx.eval(arr)
                    os.write(fd, memoryview(arr))
                    received += csz
                    del arr
                os.fsync(fd)
            finally:
                os.close(fd)
        else:
            path.touch()

        total += sz
        el = time.time() - t0
        spd = total / el / 1e9 if el > 0 else 0
        log(f"  [{i+1}/{file_count}] {name} ({sz/1e6:.1f} MB) {spd:.2f} GB/s")

    el = time.time() - t0
    spd = total / el / 1e9 if el > 0 else 0
    log(f"DONE: {file_count} files, {total/1e9:.2f} GB in {el:.1f}s ({spd:.2f} GB/s)")


def main():
    if len(sys.argv) != 3:
        print(f"Usage: {sys.argv[0]} SRC_DIR DST_DIR", file=sys.stderr)
        sys.exit(1)

    src = pathlib.Path(sys.argv[1])
    dst = pathlib.Path(sys.argv[2])

    world = mx.distributed.init()
    if world.size() < 2:
        sys.exit(1)

    # Init barrier — required for JACCL (#3149). Must use mx.ones(10), not mx.ones(1).
    barrier = mx.distributed.all_sum(mx.ones(10), group=world, stream=mx.cpu)
    mx.eval(barrier)

    try:
        if world.rank() == 0:
            if not src.is_dir():
                print(f"ERROR: Source not found: {src}", file=sys.stderr)
                sys.exit(1)
            send_directory(src, 1, world)
        elif world.rank() == 1:
            recv_directory(dst, 0, world)
    except Exception as e:
        if "protection domain" in str(e).lower():
            log(f"PD EXHAUSTION: {e}")
            sys.exit(2)
        log(f"ERROR: {e}")
        import traceback
        traceback.print_exc()
        sys.exit(1)

    gc.collect()


if __name__ == "__main__":
    main()

Wire Protocol

Sender (rank 0)                    Receiver (rank 1)
─────────────────                  ──────────────────
send(file_count: int64)     →      recv(file_count)

For each file:
  send(header: 16 bytes)    →      recv(header)
    [name_len: uint64]                [parse name_len, file_size]
    [file_size: uint64]
  send(filename: uint8[])   →      recv(filename)
  For each 64MB chunk:
    send(chunk: uint8[])    →      recv(chunk) → write to disk
                                   fsync()

Header is packed as struct.pack("<QQ", name_len, file_size) — two little-endian 64-bit unsigned integers. The last chunk of each file may be smaller than CHUNK_SIZE.

Troubleshooting

Error	Cause	Fix
RTR errno 22 (EINVAL)	Thunderbolt Bridge active	Run fix-tb-bridge.sh
RTR errno 16 (EBUSY)	Stale RDMA resources from crash	Reboot the node
Caused GPU Timeout Error	Missing stream=mx.cpu	Add CPU stream to all send/recv
Couldn't allocate protection domain	Too many init/teardown cycles	Reboot; use single-session
Shape corruption on first recv	Missing init barrier	Add all_sum(mx.ones(10)) after init
ibv_devinfo shows PORT_DOWN	No cable / wrong port / bridge active	Check cables, run fix script

Hostfile Notes

RDMA device names (rdma_en2, rdma_en5, etc.) and TB5 interface IPs are dynamically assigned every time mlx.distributed configures the mesh. They change after every reboot or mesh reconfiguration. The only stable identifiers are the 10GbE IPs in the hostfile's ips field. The hostfile must be present on all participating nodes — not just the launcher.

Limitations

• Point-to-point only: Rank 0 sends, rank 1 receives. For multi-node distribution, sequential transfers between directly-connected pairs.
• Flat directories only: Does not recurse into subdirectories.
• No resume: If interrupted, re-run the entire directory. Files are overwritten atomically (O_TRUNC).
• Reboot required for PD exhaustion: No programmatic recovery.

Hardware

• 5× Mac Studio M3 Ultra (512 GB unified memory)
• Thunderbolt 5 full mesh (4 cables per node, direct links between all pairs)
• ibv_devinfo reports 8X width at 10.0 Gbps per lane = 80 Gbps per port
• Measured ~37% utilization of theoretical single-port bandwidth

Hope this helps others working with MLX distributed and Thunderbolt 5!

[angeloskath]

Hi, you can take a look at mlx-lm and specifically mlx_lm.share which does exactly that. It saturates the hard disk easily at >5GB/s. Using all_reduce also allows sharing with more than 1 device at a time still saturating the disk speed.

Example video at https://x.com/angeloskath/status/2022502242893586670?s=20 .

Regarding the bridge issue, running mlx.distributed_config disables the bridge and sets up IPs.

PD exhaustion is interesting. We should investigate that. And I should fix issue #3149 .

[guruswami-ai]

Thanks for the quick response @angeloskath! We feel a bit silly for reinventing mlx_lm.share from scratch — just tested it on our cluster and it works beautifully at 5.2 GB/s (faster than our implementation thanks to async_eval pipelining). We should have looked harder before building our own.

That said, glad the PD exhaustion report is useful — it's reproducible if you launch ~60+ separate JACCL sessions (one per file, which is what our naive first attempt did). Only recovery is a reboot. Happy to provide more detail if it helps the investigation.

And +1 on fixing #3149 — the init barrier with all_sum(mx.ones(10)) was critical for us to get reliable point-to-point send/recv working.

One question: mlx_lm.share doesn't use stream=mx.cpu, yet we couldn't get raw send/recv to work without it (Metal GPU Timeout on the receiver, #3142). Is all_sum handled differently from send/recv internally, or does async_eval keep the Metal command buffers short enough to avoid the 5s timeout?

Thanks for building this — mlx_lm.share is exactly what we needed. The video demo is great too.

[blightbow]

Another barrier pattern you can look into for send/recv (workaround #1) is receiver initiated rendezvous. I had a great deal of luck borrowing from patterns established by MPI when I ran into issues other than data corruption. (26.3 gives us SIGABRT for that now at least)

For RIR, you would want an out of band channel that doesn't rely on RDMA. Needs to be very low latency for obvious reasons. I've played with the idea of a secondary SideChannel that is accessible from user code (i.e. a "StateChannel"), but I'm not sure how receptive the maintainers would be to that idea. Opinions?

Edit:

• Wrote that right before I went to bed..."sender initiated rendezvous" should have been receiver initiated. (for obvious reasons) Fixed.
• It's also worth noting that keeping an application accessible side channel separate is important for thread safety. Exposing SideChannel directly is not a good idea because it is not thread safe, and concurrent use between threads would corrupt the TCP socket state.

[qubitcontracting]

Finding: JACCL works WITHOUT destroying bridge0

After extensive testing on a 3-node Thunderbolt 5 cluster, we discovered that bridge0 does NOT need to be destroyed for JACCL to work. Simply assigning IP addresses to the individual Thunderbolt interfaces is sufficient -- JACCL and bridge0 can coexist.

This eliminates the need for the bridge-fix script on every boot, and avoids a critical issue where destroying bridge0 can kill USB Ethernet adapters on Mac minis.

Hardware

Node	Machine	Chip	RAM	TB Ports
Node 0	Mac mini M4 Pro	Apple M4 Pro	48 GB	3x TB5 (120 Gb/s)
Node 1	Mac Studio M3 Ultra	Apple M3 Ultra	256 GB	6x TB5 (120 Gb/s)
Node 2	Mac mini M4 Pro	Apple M4 Pro	64 GB	3x TB5 (120 Gb/s)

All three connected in a full mesh (3 Thunderbolt cables).
macOS 26.4, MLX 0.31.1, SIP disabled on all nodes.

The Problem with Destroying bridge0

The guide recommends ifconfig bridge0 destroy before every JACCL session. On Mac minis using USB 2.5GbE Ethernet adapters (common when the built-in Ethernet port is in use or insufficient), destroying bridge0 causes the USB adapter to lose connectivity. The interface disappears entirely from ifconfig and cannot be recovered without a physical replug or reboot. This makes the "destroy on every boot" approach unreliable for headless setups.

The Fix: Just Assign IPs

Instead of destroying bridge0, simply assign IP addresses to the individual Thunderbolt interfaces:

# Node 0 (Mac mini)
sudo ifconfig en4 10.0.2.2/24 up   # TB cable to Node 1

# Node 1 (Mac Studio) 
sudo ifconfig en3 10.0.1.1/24 up   # TB cable to Node 2
sudo ifconfig en4 10.0.2.1/24 up   # TB cable to Node 0

# Node 2 (Mac mini)
sudo ifconfig en4 10.0.1.2/24 up   # TB cable to Node 1
sudo ifconfig en3 10.0.3.1/24 up   # TB cable to Node 0

bridge0 remains intact. USB adapters stay connected. RDMA devices show PORT_ACTIVE:

$ ibv_devinfo | grep -E 'hca_id|state:'
hca_id: rdma_en3
        state: PORT_ACTIVE (4)
hca_id: rdma_en4
        state: PORT_ACTIVE (4)

Proof: 3-Node Simultaneous JACCL Init

With bridge0 present and active on all three nodes:

import mlx.core as mx
import os

os.environ["MLX_IBV_DEVICES"] = "/tmp/jaccl_3node.json"
os.environ["MLX_RANK"] = "0"  # 0, 1, or 2 per node
os.environ["MLX_JACCL_COORDINATOR"] = "0.0.0.0:5555"  # rank 0; others use rank 0's LAN IP

g = mx.distributed.init(backend="jaccl", strict=True)
print(f"Rank {g.rank()}: size={g.size()} SUCCESS")

Device matrix (jaccl_3node.json):

[[null, "rdma_en4", "rdma_en3"], ["rdma_en4", null, "rdma_en3"], ["rdma_en3", "rdma_en4", null]]

Output (all three nodes):

Rank 0: size=3 SUCCESS
Rank 1: size=3 SUCCESS
Rank 2: size=3 SUCCESS

Throughput Benchmark (3-Node, bridge0 present)

   Size     |   Time    | Throughput
     3.8 MB |    0.7 ms |  5,492 MB/s
    38.1 MB |    5.2 ms |  7,299 MB/s
   381.5 MB |   51.3 ms |  7,431 MB/s

7.4 GB/s sustained across 3 nodes with bridge0 intact -- matches throughput reported in this guide with bridge0 destroyed.

Real-World Validation: Distributed LLM Inference

We run this with Exo using the JACCL backend across all 3 nodes (368 GB unified memory):

• MiniMax-M2.5-8bit (226 GB): ~25 tok/s
• Qwen3-235B-A22B-Instruct-8bit (233 GB): ~14.6 tok/s, 1.6s simple responses
• The same Qwen3-235B was completely unusable on the Ring/TCP backend (3+ minute timeouts due to 94-layer pipeline parallel overhead over TCP)

Why It Works

bridge0 aggregates the TB interfaces at the network layer, but the RDMA layer (libthunderboltrdma.dylib) accesses the interfaces directly via ibv_* calls, bypassing the bridge's network stack. The individual rdma_enX devices remain addressable regardless of bridge0 state.

The errno 22 (EINVAL) that people hit when bridge0 is active likely occurs when the JACCL device matrix references bridge0's aggregated interface rather than the individual rdma_enX devices. With explicit IP assignment on individual interfaces and correct device matrix entries pointing to rdma_enX, JACCL initializes successfully.

Summary

Approach	bridge0	JACCL	USB Adapter	Boot Script
Guide (destroy)	Destroyed	Yes	No May lose connectivity	Required every boot
This finding	Present	Yes	Yes Stays connected	Just assign IPs

Note on Mac Studio TB Port Selection

On the Mac Studio M3 Ultra, we found that Receptacle 1 (front-right TB port, nearest to the Ethernet port) was unreliable for RDMA connections. Moving the cable to Receptacle 2 or 3 resolved the issue. This is a known limitation -- if you experience intermittent RDMA failures on a Mac Studio, try a different TB port before debugging further.

Patches Used in Our Setup

Our testing environment includes several patches that may be relevant if you encounter issues reproducing these results:

MLX (built from source, v0.31.1):

• DSB SY fence fix: Added __dsb(0xF) ARM64 data synchronization barrier in mlx/backend/metal/fence.cpp after f.cpu_value()[0] = count;. Without this, GPU-CPU memory coherence issues can cause hangs or corrupted outputs during distributed RDMA inference. This fix was originally from the mlx-jaccl-fix-small-recv branch by rltakashige (Exo Labs).

Exo (on custom mac-studio-cluster-fix branch):

• RDMA cycle detection fallback -- assumes full mesh when all nodes have TB interfaces (stock Exo's _graph has no RDMA edges at placement time due to a timing issue)
• is_rdma_cycle KeyError handler -- prevents crash when nodes aren't in the internal topology graph yet
• TB interface type classification -- system_info.py patch so bridge0 and TB interfaces are correctly identified as "thunderbolt" type instead of "unknown"
• Coordinator IP fallback -- allows JACCL coordinator lookup when topology subgraph has no edges
• Peer discovery retry loop -- retries EXO_PEERS connections every 5s for single-node restart recovery

The core finding (JACCL works without destroying bridge0) was verified using raw MLX Python without Exo, so the bridge0 discovery itself does not depend on any of these patches.