Ookami: MPI error opal_libevent2022_evthread_use_pthreads

[PetrKryslUCSD]

This is what I did on Ookami after I logged on. (Some more details on https://iacs-group.slack.com/archives/C016DRQ321M.)

My setup of the environment looks like this:

export JULIA_DEPOT_PATH="~/a64fx/depot"
module load julia
module load gcc/13.1.0
module load slurm
# module load mvapich2/arm22/2.3.7
module load openmpi/gcc8/4.1.2

1. I cloned the repository FinEtoolsDDParallel.jl from GitHub.
2. I activated the environment and instantiated the packages.
    julia -e 'using Pkg; Pkg.activate("."); Pkg.instantiate()'
3. I ran julia interactively to install mpiexecjl.
    julia --project=.
   I also set up the system binary: MPIPreferences.use_system_binary().
4. I ran srun to get an interactive session.
   srun -p short -n 10 --ntasks-per-node=1 --pty bash
5. I ran the example.
    cd FinEtoolsDDParallel.jl/examples/
   ~/a64fx/depot/bin/mpiexecjl -n 4 julia --project=. heat/Poisson2D_cg_mpi_driver.jl

After several minutes, the job was terminated. The error message is below.
Note well: On my laptop this example runs to completion in ~70 seconds.

julia: symbol lookup error: /lustre/home/pkrysl/a64fx/depot/artifacts/58dcf187642cdfbafb3581993ca3d8de565acc78/lib/openmpi/mca_pmix_pmix3x.so: undefined symbol: opal_libevent2022_evthread_use_pthreads
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
[1972752] signal (15): Terminated
in expression starting at /lustre/home/pkrysl/a64fx/FinEtoolsDDParallel.jl/examples/heat/Poisson2D_cg_mpi_driver.jl:160
_ZN12_GLOBAL__N_117InterleavedAccess13runOnFunctionERN4llvm8FunctionE at /lustre/software/julia/julia-1.10.3/lib/julia/libLLVM-15jl.so (unknown line)
_ZN12_GLOBAL__N_117InterleavedAccess13runOnFunctionERN4llvm8FunctionE at /lustre/software/julia/julia-1.10.3/lib/julia/libLLVM-15jl.so (unknown line)
unknown function (ip: (nil))
Allocations: 14621931 (Pool: 14605380; Big: 16551); GC: 19
julia: symbol lookup error: /lustre/home/pkrysl/a64fx/depot/artifacts/58dcf187642cdfbafb3581993ca3d8de565acc78/lib/openmpi/mca_pmix_pmix3x.so: undefined symbol: opal_libevent2022_evthread_use_pthreads
julia: symbol lookup error: /lustre/home/pkrysl/a64fx/depot/artifacts/58dcf187642cdfbafb3581993ca3d8de565acc78/lib/openmpi/mca_pmix_pmix3x.so: undefined symbol: opal_libevent2022_evthread_use_pthreads
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
  Process name: [[7833,1],1]
  Exit code:    127

[PetrKryslUCSD]

There is something wrong with the system mpi binary. Doing julia --project -e 'using MPIPreferences; MPIPreferences.use_system_binary()' leads to the above error; doing julia --project -e 'using MPIPreferences; MPIPreferences.use_jll_binary("OpenMPI_jll")' does not.

[luraess]

Seems something may go indeed wrong then when trying to hook into system libs. Maybe look into https://github.com/giordano/julia-on-ookami as IIRC @giordano did quite some extensive testing and use of that machine.

[giordano]

I'm a bit late to the party here, but I think I have a solution for your problem. In short, this isn't an issue in MPI.jl per se (which if configured correctly to use system MPI would do the right thing), but this happens if using other packages which would load JLLs which link to MPI, and when they see the system MPI has OpenMPI ABI they go and load the OpenMPI_jll library, because we have no notion of "system ABI" in the JLL multiplexing mechanism. A common culprit here is HDF5_jll, when I looked into this issue last year I traced the issue to that package.

Now, how to address it.

$ module load openmpi/gcc8/4.1.2
Loading openmpi/gcc8/4.1.2
  Loading requirement: ucx/1.11.2
$ module load julia
$ julia --project -q
(openmpi) pkg> add MPIPreferences, MPI
   Resolving package versions...
    Updating `~/tmp/openmpi/Project.toml`
  [da04e1cc] + MPI v0.20.22
  [3da0fdf6] + MPIPreferences v0.1.11
  [...]

julia> using MPIPreferences

julia> MPIPreferences.use_system_binary()
┌ Info: MPI implementation identified
│   libmpi = "libmpi"
│   version_string = "Open MPI v4.1.2rc4, package: Open MPI decarlson@fj100 Distribution, ident: 4.1.2rc4, repo rev: 2022-04-05, Unreleased developer copy\0"
│   impl = "OpenMPI"
│   version = v"4.1.2-rc4"
└   abi = "OpenMPI"
┌ Info: MPIPreferences changed
│   binary = "system"
│   libmpi = "libmpi"
│   abi = "OpenMPI"
│   mpiexec = "mpiexec"
│   preloads = Any[]
└   preloads_env_switch = nothing

julia> exit()

$ julia --project -q
julia> using MPI, Libdl

julia> filter(contains("libmpi"), dllist())
1-element Vector{String}:
 "/lustre/software/openmpi/gcc8/4.1.2-rocky/lib/libmpi.so"

We followed the documentation and MPI.jl loads only system libmpi, so far so good, no surprises. As I anticipated above, the problem comes when you load another package which links to libmpi:

(openmpi) pkg> add HDF5_jll
   [...]

julia> using MPI, HDF5_jll, Libdl

julia> filter(contains("libmpi"), dllist())
2-element Vector{String}:
 "/lustre/software/openmpi/gcc8/4.1.2-rocky/lib/libmpi.so"
 "/lustre/home/mosgiordano/.julia" ⋯ 34 bytes ⋯ "43e725d0e5f9dfa0a/lib/libmpi.so"

Now we have a problem: when loading both MPI and HDF5_jll two different libmpis are loaded, which is the cause of the segfault you were observing: MPI.jl initialises system libmpi, but if you use some hdf5 functionalities they end up calling libmpi from the JLL, which wasn't initialised, and then we go boom.

Luckily, for the case where system OpenMPI has the same ABI as the libmpi in OpenMPI_jll (i.e. also same soname, libmpi.so.40) we have an option: point libmpi in OpenMPI_jll to system libmpi. In the LocalPreferences.toml file created by MPIPreferences.toml in your environment directory add the following section, below the [MPIPreferences] one:

[OpenMPI_jll]
libmpi_path = "/lustre/software/openmpi/gcc8/4.1.2-rocky/lib/libmpi.so"
mpiexec_path = "/lustre/software/openmpi/gcc8/4.1.2-rocky/bin/mpiexec" # probably not necessary, but we set this for good measure

The paths above are the paths of libmpi and mpiexec for the openmpi module you want to use (here I used the paths for the openmpi/gcc8/4.1.2 module on Ookami, other people reading this would need to adapt to their case). Then, also add the OpenMPI_jll package to the environment (]add OpenMPI_jll) and restart the Julia session. If all goes well, at the restart you should see:

julia> using MPI, HDF5_jll, Libdl

julia> filter(contains("libmpi"), dllist())
1-element Vector{String}:
 "/lustre/software/openmpi/gcc8/4.1.2-rocky/lib/libmpi.so"

Now when loading both MPI and HDF5_jll we're back to the ideal situation of having a single libmpi loaded at all times.

I won't say this is a great solution, because it requires duplicating some information MPIPreferences already tries to put there, plus some manual fiddling with the LocalPreferences.toml file. Also, I considered automating this in MPIPreferences, but this won't work in general if system libmpi has OpenMPI ABI but not the soname libmpi.so.40. For that case the only option I see is to use mpitrampoline.

Hopefully, life will be better in a couple of years when the new standard MPI ABI will be more widespread, because facilities will quickly adopt it, right?