Installation issue: py-scipy / spack-stack @ develop fails to configure with oneAPI @ 2024.2.1 on hercules and orion

[rickgrubin-noaa]

Describe the issue

py-scipy fails to configure / build / install when spec'd for oneAPI @ 2024.2.1 for spack-stack @ develop (updated with two PRs)

py-scipy @ 1.14.1 spec: py-scipy.spec.txt

This failure prevents a unified-dev env from building / installing.

Pertinent error output:

• log.install.txt
• build-stage.txt

The failure appears identical to that which is reported in Installation issue: [email protected] with Intel Classic compilers #48243 and fixed in Add conflict for [email protected]: with Intel Classic #48251, despite that PR being applied.

To Reproduce

spack-stack @ develop with changes incorporated from:

Avoid linking to libirc.so in spack (parallel-netcdf), turn off crypt variant for Python, and update Orion site config to fix tar issue #1435

and

Bug fix in parallel-netcdf to avoid linking to libirc.so AND cherry-pick spack develop PR 48251 (conflict Intel Classic with [email protected]) #495.

The fix from Add conflict for [email protected]: with Intel Classic #48251 is already present in the JCSDA spack specified in spack-stack @ develop / .gitmodules -- the build-stage output clearly shows that a valid oneAPI compiler is found.

Configuration files:

• site-compilers.yaml site-compilers.yaml.txt

• site-packages_oneapi.yaml site-packages_oneapi.yaml.txt

• site-packages.yaml site-packages.yaml.txt

• common-packages_oneapi.yaml common-packages_oneapi.yaml.txt

• common-packages.yaml common-packages.yaml.txt

Expected behavior

py-scipy configures, builds, and installs with oneAPI @ 2024.2.1 compilers (as a preface to spack-stack @ release/1.9.0)

System:

tier1 EPIC hosts hercules (linux-rocky9-icelake) and orion (linux-rocky9-skylake_avx512)

Additional context

A test env to install only py-scipy % gcc (and dependencies) successfully configures / builds / installs.

py-scipy @ 1.14.1 % gcc spec: py-scipy-spec.gcc.txt

Given this success, I'm unclear if something is misconfigured or if this is a bug somewhere.

[climbfuji]

I see the meson error in the install log. Can you also upload the meson log file, please?

/work/noaa/epic/role-epic/spack-stack-testing/spack-stack-1.9.0/cache/build_stage/spack-stage-py-scipy-1.14.1-hsd4od3zcazqqupk5lhpinnr3y4lig2g/spack-src/build/meson-logs/meson-log.txt

[rickgrubin-noaa]

I see the meson error in the install log. Can you also upload the meson log file, please?

That's referenced as build-stage.txt in the attached files.

[climbfuji]

I see it now, thanks. I am wondering why this is a problem with icx. What version is the backend GNU compiler?

[rickgrubin-noaa]

I see it now, thanks. I am wondering why this is a problem with icx. What version is the backend GNU compiler?

System-provided: 11.3.1 which ought to be adequate. The meson-build output didn't care for GNU ld version 2.35.2-24 as a linker; the cfg for icx says, if it's helpful:

-Wl,-rpath,/apps/spack-managed-x86_64_v3-v1.0/gcc-11.3.1/intel-oneapi-compilers-2024.2.1-podbez65l57ms4uba527kg7pomxk5y3m/compiler/2024.2/lib --gcc-toolchain=/usr

---

Module available to use, if desired: 14.2.0

The successful py-scipy only env build with %gcc was with 14.2.0, so it makes sense to reconfigure a unified-dev build that sets the prepend_path vars for PATH / LD_LIBRARY_PATH / CPATH to use those available with gcc/14.2.0. It's time-consuming because py-scipy is one of the last packages built for a unified-dev build. Then again, the icx cfg settings may render this moot.

Separate but potentially related question: does it make sense to require %gcc for building glibc ? Doesn't seem to have been at issue before, as far as I can tell, but am curious.

[climbfuji]

gcc@11 is more than enough for C++-17 - I never tried using gcc@14 as backend for Intel oneAPI.

Not sure about glibc either. We don't need to build this with gcc on any of the other platforms.

I am pretty sure I can build py-scipy on the NRL machines with oneAPI 2024 (icx, icpx, ifort) using gcc@11 as backend.

[rickgrubin-noaa]

gcc@11 is more than enough for C++-17 - I never tried using gcc@14 as backend for Intel oneAPI.

That's why I chose to use the system-supplied gcc@11 -- it met the requirements.

Not sure about glibc either. We don't need to build this with gcc on any of the other platforms.

Understood, was just curious.

I am pretty sure I can build py-scipy on the NRL machines with oneAPI 2024 (icx, icpx, ifort) using gcc@11 as backend.

Hence raising this issue -- I suspect it's probably not showing up elsewhere, and am concerned that it's a config issue / error on my part. What holds me back a bit there is the meson-build error about GNU ld.

[rickgrubin-noaa]

py-scipy installs with oneAPI @ 2024.2.1 on other hosts (ursa, for example). That particular installation was built with intel-oneapi-mkl.

On hercules and orion, I was building with openblas. This works with GNU compilers, but not with oneAPI compilers.

Building a test env for just py-scipy, and using --keep-stage to retain the build dirs, portions of the meson build dirs for py-scipy are incomplete, when compared to a build_cache dir on ursa. In particular, the directory that contains test C / C++ / Fortran files to verify the compilers, as well as ancillary config files, is empty, as is a dir (meson-info) that is clearly written when configuring, as it contains host-specific pathing to compilers and such.

The error that shows up in the meson-log.txt file happens because there's no test file to compile. This is quite odd.

Trying again (standalone py-scipy env) with intel-oneapi-mkl instead of openblas yielded the same negative result.

For whatever reason(s), py-scipy is not configuring correctly / configuring incompletely to build / install. There are no ninja-generated make / etc. files -- files that exist on other hosts with successful builds.

[rickgrubin-noaa]

Still see this behavior only on hercules and orion, and nowhere else -- be it EPIC hosts, or elsewhere.

Closing as it can be solved with:

  py-scipy:
    require:
      '%gcc'

in configs/sites/tier1/hercules/packages_oneapi.yaml