Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build / test UFS WM with Intel oneAPI v2024.2.1 (LLVM)-based stacks #1390

Open
rickgrubin-noaa opened this issue Nov 27, 2024 · 5 comments
Open

Build / test UFS WM with Intel oneAPI v2024.2.1 (LLVM)-based stacks #1390

rickgrubin-noaa opened this issue Nov 27, 2024 · 5 comments
Assignees
@rickgrubin-noaa
Labels
OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center

Comments

@rickgrubin-noaa
Copy link
Collaborator

Description

Build and test UFS WM against spack-stack@release/1.8.0 built with Intel oneAPI v2024.2.1 (LLVM) compilers.

Solution

stack environments

hera: /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs

  • ifx

    • ue-llvm-ifx-2024.2.1
    • gsi-llvm-ifx-2024.2.1

  • ifort

    • ue-llvm-ifort-2024.2.1
    • gsi-llvm-ifort-2024.2.1

Will also require addon environments for each compiler for a known-to-work combination of MAPL / ESMF.

Once the MAPL / ESMF addon envs are ready, steps to load required modules, e.g. for the ifx compiler:

module purge
module use /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/modulefiles/Core
module load stack-oneapi/2024.2.1
module load stack-intel-oneapi-mpi/2021.13
module load ufs-weather-model-env/1.0.0

Build and test UFS WM with appropriate changes to modules loaded to reflect the above stacks.

Additional Context

This issue is to track the UFS WM issue Build / test UFS WM with Intel oneAPI v2024.2.1 (LLVM) compilers #2513
@climbfuji climbfuji added the OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center label Nov 27, 2024
@climbfuji climbfuji moved this to In Progress in spack-stack-1.9.0 (2024 Q4) Nov 27, 2024
@rickgrubin-tomorrow
Copy link

[ Ref: internal EPIC ticket; note also that results / path below are specific to hera ]

UFS WM Regression Testing results

ifort

ifx

For this case: [email protected] (failed compile (DEBUG))

[...]
[100%] Linking Fortran executable ufs_model
/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/cmake-3.27.9-ihjoyd3/bin/cmake -E cmake_link_script CMakeFiles/ufs_model.dir/link.txt --verbose=1
/apps/oneapi/mpi/2021.13/bin/mpifc -fiopenmp  -g -traceback -fpp -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -align array64byte -qno-opt-dynamic-align -diag-disable 5462 -diag-disable 7712 -diag-disable 8291 -diag-disable 5194 -g -O0 -check -check noarg_temp_created -check nopointer -warn -warn noerrors -fp-stack-check -fstack-protector-all -fpe0 -debug minimal -ftrapuv -init=snan,arrays CMakeFiles/ufs_model.dir/driver/UFS.F90.o -o ufs_model 
[...]
-L/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/parallelio-2.6.2-prs7utx/lib -lrt -ldl -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lm -lpioc -m64 -mcmodel=small -pthread -threads
[...]

ld: cannot find -lpioc
ld: attempted static link of dynamic object `/apps/oneapi/compiler/2024.2/lib/libiomp5.so'

and

% file /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/parallelio-2.6.2-prs7utx/lib/libpioc.so
/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/parallelio-2.6.2-prs7utx/lib/libpioc.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

% ldd /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/parallelio-2.6.2-prs7utx/lib/libpioc.so
	linux-vdso.so.1 (0x00007ffea02e0000)
	libnetcdf.so.19 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/netcdf-c-4.9.2-ulez3hv/lib/libnetcdf.so.19 (0x00007f8c050b6000)
	libpnetcdf.so.4 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/parallel-netcdf-1.12.3-3olk4vh/lib/libpnetcdf.so.4 (0x00007f8c0445e000)
	libmpifort.so.12 => /apps/oneapi/mpi/2021.13/lib/libmpifort.so.12 (0x00007f8c040b2000)
	libmpi.so.12 => /apps/oneapi/mpi/2021.13/lib/libmpi.so.12 (0x00007f8bfa277000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f8bfa073000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f8bf9e6b000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8bf9c4b000)
	libsvml.so => /apps/oneapi/compiler/2024.2/lib/libsvml.so (0x00007f8bf8607000)
	libirng.so => /apps/oneapi/compiler/2024.2/lib/libirng.so (0x00007f8c05869000)
	libimf.so => /apps/oneapi/compiler/2024.2/lib/libimf.so (0x00007f8bf81fb000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f8bf7e79000)
	libintlc.so.5 => /apps/oneapi/compiler/2024.2/lib/libintlc.so.5 (0x00007f8c05806000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f8bf7aa3000)
	libhdf5_hl.so.310 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/hdf5-1.14.3-5szfql6/lib/libhdf5_hl.so.310 (0x00007f8bf7880000)
	libhdf5.so.310 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/hdf5-1.14.3-5szfql6/lib/libhdf5.so.310 (0x00007f8bf71a9000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f8bf6f98000)
	libzstd.so.1 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/zstd-1.5.2-clennzi/lib/libzstd.so.1 (0x00007f8bf6cae000)
	libblosc.so.1 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/c-blosc-1.21.5-rwudonl/lib64/libblosc.so.1 (0x00007f8bf6a84000)
	libxml2.so.2 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/libxml2-2.10.3-tnoi22x/lib/libxml2.so.2 (0x00007f8bf66eb000)
	libcurl.so.4 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/curl-8.7.1-ilyildl/lib/libcurl.so.4 (0x00007f8bf6427000)
	libifport.so.5 => /apps/oneapi/compiler/2024.2/lib/libifport.so.5 (0x00007f8c057d8000)
	libifcoremt.so.5 => /apps/oneapi/compiler/2024.2/lib/libifcoremt.so.5 (0x00007f8bf62bb000)
	libirc.so => /apps/oneapi/compiler/2024.2/lib/libirc.so (0x00007f8bf625a000)
	libstdc++.so.6 => /apps/gnu/gcc-9.2.0/lib64/libstdc++.so.6 (0x00007f8bf5e7b000)
	libgcc_s.so.1 => /apps/gnu/gcc-9.2.0/lib64/libgcc_s.so.1 (0x00007f8bf5c63000)
	libmpicxx.so.12 => /apps/oneapi/mpi/2021.13/lib/libmpicxx.so.12 (0x00007f8bf5a43000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8c0574a000)
	liblz4.so.1 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/lz4-1.9.4-mt5apsb/lib/liblz4.so.1 (0x00007f8bf5816000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f8bf55ef000)
	libnghttp2.so.14 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/nghttp2-1.57.0-w273rt4/lib/libnghttp2.so.14 (0x00007f8bf53bf000)
	libssl.so.3 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/openssl-3.3.0-2cqpman/lib64/libssl.so.3 (0x00007f8bf50c4000)
	libcrypto.so.3 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/openssl-3.3.0-2cqpman/lib64/libcrypto.so.3 (0x00007f8bf4aa0000)



% file /apps/oneapi/compiler/2024.2/lib/libiomp5.so
/apps/oneapi/compiler/2024.2/lib/libiomp5.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

%  ldd /apps/oneapi/compiler/2024.2/lib/libiomp5.so
	linux-vdso.so.1 (0x00007ffe8988d000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f888c09f000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f888be9b000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f888bc7b000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f888b8a5000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f888c68f000)

For the simple test run (control_c48), the DEBUG settings are:

COMPILE | atm_dyn32 | intelllvm | -DAPP=ATM -DDEBUG=ON -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn,FV3_GFS_v17_p8_ugwpv1 -D32BIT=ON | | fv3 |
RUN | control_c48                                       |                                      | baseline |

@rickgrubin-noaa rickgrubin-noaa mentioned this issue Jan 23, 2025
Open
12 tasks
@ulmononian
Copy link
Collaborator

just to note, we've been testing 1.8.0 w/ oneapi compilers (icx,icpx,ifort) on ursa and hit errors with Cmake not finding MPI_C when building in ATM mode: ufs-community/ufs-weather-model#2471 (comment).

oddly, we hit the same error with cmake (MPI_C not being found) when trying to compile ufs-wm in S2SWA mode using spack-stack/1.6.0 w/ icx/icpx/ifort. however, ATM mode builds fine with this 1.6.0 stack.

-- Could NOT find MPI_C (missing: MPI_C_WORKS)

@climbfuji
Copy link
Collaborator

just to note, we've been testing 1.8.0 w/ oneapi compilers (icx,icpx,ifort) on ursa and hit errors with Cmake not finding MPI_C when building in ATM mode: ufs-community/ufs-weather-model#2471 (comment).

oddly, we hit the same error with cmake (MPI_C not being found) when trying to compile ufs-wm in S2SWA mode using spack-stack/1.6.0 w/ icx/icpx/ifort. however, ATM mode builds fine with this 1.6.0 stack.

-- Could NOT find MPI_C (missing: MPI_C_WORKS)

I bet this has to do with the Intel ifcore libc missing symbols.

@ulmononian
Copy link
Collaborator

just to note, we've been testing 1.8.0 w/ oneapi compilers (icx,icpx,ifort) on ursa and hit errors with Cmake not finding MPI_C when building in ATM mode: ufs-community/ufs-weather-model#2471 (comment).

oddly, we hit the same error with cmake (MPI_C not being found) when trying to compile ufs-wm in S2SWA mode using spack-stack/1.6.0 w/ icx/icpx/ifort. however, ATM mode builds fine with this 1.6.0 stack.

-- Could NOT find MPI_C (missing: MPI_C_WORKS)

I bet this has to do with the Intel ifcore libc missing symbols.

@climbfuji is there a fix for that that i can test? (apologies in advance if there is an issue for this that is not on my radar)

@climbfuji
Copy link
Collaborator

My guess is that this is the Intel issue we've been talking about for a year, but I am not sure. You need to run cmake with debug or trace output enabled to see what the error is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rickgrubin-noaa rickgrubin-noaa

Labels
OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center
Projects
spack-stack-1.9.0 (2024 Q4)
Status: In Progress
Milestone
No milestone
Development

No branches or pull requests
4 participants
@climbfuji @ulmononian @rickgrubin-tomorrow @rickgrubin-noaa