Benchmark cluster of Mac Studios

[geerlingguy]

After completing #89, I'd like to cluster a few Macs together and see what kind of performance I can get. Networking on the Macs is a little odd, but should work I hope.

Right now I'm getting:

TASK [Run the benchmark.] **********************************************************************************************
fatal: [mac-studio-1.local]: FAILED! => 
    changed: true
    cmd:
    - /opt/top500/mpich/bin/mpirun
    - -f
    - cluster-hosts
    - ./xhpl
    delta: '0:00:00.799710'
    end: '2025-12-08 16:21:25.980046'
    msg: non-zero return code
    rc: 93
    start: '2025-12-08 16:21:25.180336'
    stderr: |-
        Fatal error in internal_Init: Unknown error class, error stack:
        internal_Init(70)...........................: MPI_Init(argc=0x16afb7154, argv=0x16afb7148) failed
        MPII_Init_thread(327).......................:
        MPIR_nodeid_init(457).......................:
        MPIR_Allgather_impl(2216)...................:
        MPIR_Allgather_allcomm_auto(2136)...........:
        MPIR_Allgather_intra_recursive_doubling(85).:
        MPIC_Sendrecv(263)..........................:
        MPIC_Wait(90)...............................:
        MPIR_Wait(751)..............................:
        MPIR_Wait_state(708)........................:
        MPIDI_CH3i_Progress_wait(187)...............: an error occurred while handling an event returned by MPIDI_CH3I_Sock_Wait()
        MPIDI_CH3I_Progress_handle_sock_event(572)..:
        MPIDI_CH3_Sockconn_handle_connect_event(518): [ch3:sock] failed to connect to remote process
        MPIDI_CH3I_Socki_handle_connect(3787).......: connection failure (set=0,sock=1,errno=61:Connection refused)
    stderr_lines: <omitted>
    stdout: ''
    stdout_lines: <omitted>

Same thing when manually running /opt/top500/mpich/bin/mpirun -f cluster-hosts ./xhpl.

[geerlingguy]

I've tried using mDNS hostnames instead of IP addresses in cluster-hosts, but am getting:

[proxy:1@mac-studio-2.local] HYDU_sock_connect (lib/utils/sock.c:142): unable to connect from "mac-studio-2.local" to "mac-studio-1.local" (Invalid argument)
[proxy:1@mac-studio-2.local] main (proxy/pmip.c:101): unable to connect to server mac-studio-1.local at port 49707 (check for firewalls!)
[mpiexec@mac-studio-1.local] ui_cmd_cb (mpiexec/pmiserv_pmci.c:51): Launch proxy failed.
[mpiexec@mac-studio-1.local] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[mpiexec@mac-studio-1.local] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:180): error waiting for event
[mpiexec@mac-studio-1.local] main (mpiexec/mpiexec.c:260): process manager error waiting for completion

So it seems like maybe /etc/hosts issues or reverse-SSH-connection issues.

[geerlingguy]

Testing a very basic program test.c:

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
    MPI_Init(NULL, NULL);
    int rank;
    int world;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &world);
    printf("Hello: rank %d, world: %d\n",rank, world);
    MPI_Finalize();
}

Compile with /opt/top500/mpich/bin/mpicc -o test ./test.c, then:

/opt/top500/mpich/bin/mpirun -np 2 ./test
Hello: rank 0, world: 2
Hello: rank 1, world: 2

Found on https://stackoverflow.com/a/42780322/100134

jgeerling@mac-studio-1 top500 % /opt/top500/mpich/bin/mpirun -f cluster-hosts ./test 
[proxy:1@mac-studio-2.local] HYDU_create_process (lib/utils/launch.c:73): execvp error on file ./test (No such file or directory)
...
[proxy:2@mac-studio-3.local] HYDU_create_process (lib/utils/launch.c:73): execvp error on file ./test (No such file or directory)
...

I compiled test on all the Macs and retested...

% /opt/top500/mpich/bin/mpirun -f cluster-hosts ./test
Fatal error in internal_Init: Unknown error class, error stack:
internal_Init(70)...........................: MPI_Init(argc=0x0, argv=0x0) failed
MPII_Init_thread(327).......................: 
MPIR_nodeid_init(457).......................: 
MPIR_Allgather_impl(2216)...................: 
MPIR_Allgather_allcomm_auto(2136)...........: 
MPIR_Allgather_intra_recursive_doubling(85).: 
MPIC_Sendrecv(263)..........................: 
MPIC_Wait(90)...............................: 
MPIR_Wait(751)..............................: 
MPIR_Wait_state(708)........................: 
MPIDI_CH3i_Progress_wait(187)...............: an error occurred while handling an event returned by MPIDI_CH3I_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(572)..: 
MPIDI_CH3_Sockconn_handle_connect_event(518): [ch3:sock] failed to connect to remote process
MPIDI_CH3I_Socki_handle_connect(3787).......: connection failure (set=0,sock=1,errno=61:Connection refused)

So it's definitely an inter-node communication issue. I even plugged into wired Ethernet and disabled WiFi (thinking maybe WiFi was causing shenanigans, I just wanted a quick networked test environment), and it's still not working.

I also tried the simplest cluster communication test, and that actually works:

% /opt/top500/mpich/bin/mpirun -f cluster-hosts hostname
mac-studio-1.local
mac-studio-1.local
mac-studio-1.local
...
mac-studio-4.local
mac-studio-4.local
...
mac-studio-2.local
...
mac-studio-3.local

As found in the debugging steps here.

I have confirmed I can SSH to each node from each node as jgeerling without any issues or needing to accept any hostkeys. It's like there's something system level blocking me. Firewall is off on all the Macs.

[geerlingguy]

Oh hey! After reading through some of the mailing list debugging, I found that some people suggested using tcp instead of sockets... and I found the default option is now ch3:nemesis (when did that change, sheesh...).

So I removed mpich from all nodes, recompiled with --with-device=ch3:nemesis, and am running again... and look at this!

We're cooking now.

[geerlingguy]

Preliminary run, this is with all 4x M3 Ultra Mac Pro nodes interconnected via TB5, which is not the most efficient setup as they are idling (as a cluster) around 180W:

3,504.1 Gflops at 790W, for 4.44 Gflops/W

================================================================================
HPLinpack 2.3  --  High-Performance Linpack benchmark  --   December 2, 2018
Written by A. Petitet and R. Clint Whaley,  Innovative Computing Laboratory, UTK
Modified by Piotr Luszczek, Innovative Computing Laboratory, UTK
Modified by Julien Langou, University of Colorado Denver
================================================================================

An explanation of the input/output parameters follows:
T/V    : Wall time / encoded variant.
N      : The order of the coefficient matrix A.
NB     : The partitioning blocking factor.
P      : The number of process rows.
Q      : The number of process columns.
Time   : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.

The following parameter values will be used:

N      :  288953
NB     :     256
PMAP   : Row-major process mapping
P      :       8
Q      :      16
PFACT  :   Right
NBMIN  :       4
NDIV   :       2
RFACT  :   Crout
BCAST  :  1ringM
DEPTH  :       1
SWAP   : Mix (threshold = 64)
L1     : transposed form
U      : transposed form
EQUIL  : yes
ALIGN  : 8 double precision words

--------------------------------------------------------------------------------

- The matrix A is randomly generated for each test.
- The following scaled residual check will be computed:
      ||Ax-b||_oo / ( eps * ( || x ||_oo * || A ||_oo + || b ||_oo ) * N )
- The relative machine precision (eps) is taken to be               1.110223e-16
- Computational tests pass if scaled residuals are less than                16.0

================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR11C2R4      288953   256     8    16            4589.98             3.5041e+03
HPL_pdgesv() start time Mon Dec  8 18:16:16 2025

HPL_pdgesv() end time   Mon Dec  8 19:32:46 2025

--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=   1.79854583e-03 ...... PASSED
================================================================================

Finished      1 tests with the following results:
              1 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.
--------------------------------------------------------------------------------

End of Tests.
================================================================================

Power usage graph:

[geerlingguy]

Re-ran again today, with everything stripped down:

3,731.1 Gflops at 780W, for 4.78 Gflops/W

================================================================================
HPLinpack 2.3  --  High-Performance Linpack benchmark  --   December 2, 2018
Written by A. Petitet and R. Clint Whaley,  Innovative Computing Laboratory, UTK
Modified by Piotr Luszczek, Innovative Computing Laboratory, UTK
Modified by Julien Langou, University of Colorado Denver
================================================================================

An explanation of the input/output parameters follows:
T/V    : Wall time / encoded variant.
N      : The order of the coefficient matrix A.
NB     : The partitioning blocking factor.
P      : The number of process rows.
Q      : The number of process columns.
Time   : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.

The following parameter values will be used:

N      :  288953
NB     :     256
PMAP   : Row-major process mapping
P      :       8
Q      :      16
PFACT  :   Right
NBMIN  :       4
NDIV   :       2
RFACT  :   Crout
BCAST  :  1ringM
DEPTH  :       1
SWAP   : Mix (threshold = 64)
L1     : transposed form
U      : transposed form
EQUIL  : yes
ALIGN  : 8 double precision words

--------------------------------------------------------------------------------

- The matrix A is randomly generated for each test.
- The following scaled residual check will be computed:
      ||Ax-b||_oo / ( eps * ( || x ||_oo * || A ||_oo + || b ||_oo ) * N )
- The relative machine precision (eps) is taken to be               1.110223e-16
- Computational tests pass if scaled residuals are less than                16.0

================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR11C2R4      288953   256     8    16            4310.73             3.7311e+03
HPL_pdgesv() start time Tue Dec  9 09:45:23 2025

HPL_pdgesv() end time   Tue Dec  9 10:57:14 2025

--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=   1.79854583e-03 ...... PASSED
================================================================================

Finished      1 tests with the following results:
              1 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.
--------------------------------------------------------------------------------

End of Tests.
================================================================================

[geerlingguy]

I tried getting HPL running over Thunderbolt, with every node connected to every other node, but I realized getting the communication set up through mpirun without a switch and global addressing is... problematic.

So I narrowed my sights and focused on just using two nodes instead of all four.

I set up mac-studio-1.local and mac-studio-2.local to communicate with each other, and ran:

jgeerling@mac-studio-1 top500 % /opt/top500/mpich/bin/mpirun -f cluster-hosts ./xhpl
================================================================================
HPLinpack 2.3  --  High-Performance Linpack benchmark  --   December 2, 2018
Written by A. Petitet and R. Clint Whaley,  Innovative Computing Laboratory, UTK
Modified by Piotr Luszczek, Innovative Computing Laboratory, UTK
Modified by Julien Langou, University of Colorado Denver
================================================================================

An explanation of the input/output parameters follows:
T/V    : Wall time / encoded variant.
N      : The order of the coefficient matrix A.
NB     : The partitioning blocking factor.
P      : The number of process rows.
Q      : The number of process columns.
Time   : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.

The following parameter values will be used:

N      :  288953 
NB     :     256 
PMAP   : Row-major process mapping
P      :       4 
Q      :       8 
PFACT  :   Right 
NBMIN  :       4 
NDIV   :       2 
RFACT  :   Crout 
BCAST  :  1ringM 
DEPTH  :       1 
SWAP   : Mix (threshold = 64)
L1     : transposed form
U      : transposed form
EQUIL  : yes
ALIGN  : 8 double precision words

--------------------------------------------------------------------------------

- The matrix A is randomly generated for each test.
- The following scaled residual check will be computed:
      ||Ax-b||_oo / ( eps * ( || x ||_oo * || A ||_oo + || b ||_oo ) * N )
- The relative machine precision (eps) is taken to be               1.110223e-16
- Computational tests pass if scaled residuals are less than                16.0


  [Restored Dec 13, 2025 at 11:18:10 PM]
Last login: Sat Dec 13 23:18:07 on console

As you can see at the end, the system force-rebooted. I'm not sure if the Thunderbolt networking enjoys having HPL running through it! I'll give it another try though, lol.

[geerlingguy]

I tried a few more times, and every time it would completely lock up the system... for now I'll put that on hold. I may try Thunderbolt again if I can figure out MPX and mpirun together, for more automated networking...