ATS-9990: SPEC - expanse/0.17.3.b - User requested Intel TBB installation

[mkandes]

https://access-ci.atlassian.net/browse/ATS-9980

[mkandes]

Before creating a new featture branch to deploy Intel TBB, you need to resync the deployment branch of your fork with the sdsc/spack upstream.

[mkandes@login01 spack]$ pwd
/home/mkandes/software/spack/repositories/mkandes/spack
[mkandes@login01 spack]$ git log -1
commit 2eccf0894939f5e1f34d8bde648884b6ad5f0d79 (HEAD -> sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf, origin/sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf)
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Fri Aug 16 11:33:00 2024 -0700

    Add nvhpc@21.9 % gcc@8.5.0 + netcdf-* to exp/0.17.3/cpu/b
    
    We are deploying nvhpc@21.9 to exp/0.17.3/cpu/b as a compiler option to
    support a specific use case [1], where the end user can currently only
    compile their custom code with PGI-based compilers and a set of
    compatible NetCDF libraries.
    
    [1] https://github.com/sdsc/spack/issues/125
[mkandes@login01 spack]$ git branch
  sdsc-0.17.3
  sdsc-0.17.3-gh-124-inst-exp-cpu-dev
* sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf
[mkandes@login01 spack]$ git checkout sdsc-0.17.3
Switched to branch 'sdsc-0.17.3'
Your branch is up to date with 'origin/sdsc-0.17.3'.
[mkandes@login01 spack]$

[mkandes]

Resyncing from upstream ...

[mkandes@login01 spack]$ git pull upstream sdsc-0.17.3
From https://github.com/sdsc/spack
 * branch                  sdsc-0.17.3 -> FETCH_HEAD
Updating 3cbfd1ba45..a3029ae4c3
Fast-forward
 .../cpu/b/specs/cmake@3.21.4.o33147784.exp-15-56   | 653 ++++++++++++++++++
 .../expanse/0.17.3/cpu/b/specs/cmake@3.21.4.sh     |  68 ++
 .../gcc@10.2.0/fftw@3.3.10-omp.o33150584.exp-15-56 | 662 ++++++++++++++++++
 .../cpu/b/specs/gcc@10.2.0/fftw@3.3.10-omp.sh      |   5 +-
 .../0.17.3/cpu/b/specs/gcc@10.2.0/fftw@3.3.10.sh   |   2 +-
 .../fftw@3.3.10-omp.o33160030.exp-15-56            | 742 +++++++++++++++++++++
 .../gcc@10.2.0/mvapich2@2.3.7/fftw@3.3.10-omp.sh   |  81 +++
 .../specs/gcc@10.2.0/mvapich2@2.3.7/fftw@3.3.10.sh |   2 +-
 .../specs/gcc@10.2.0/openmpi@4.1.3/fftw@3.3.10.sh  |   2 +-
 .../cpu/b/specs/nvhpc@21.9.o33146549.exp-15-56     | 643 ++++++++++++++++++
 .../sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9.sh  |  72 ++
 .../nvhpc@21.9/hdf5@1.10.7.o33147827.exp-15-56     | 695 +++++++++++++++++++
 .../0.17.3/cpu/b/specs/nvhpc@21.9/hdf5@1.10.7.sh   |  68 ++
 .../nvhpc@21.9/netcdf-c@4.8.1.o33148039.exp-15-56  | 704 +++++++++++++++++++
 .../cpu/b/specs/nvhpc@21.9/netcdf-c@4.8.1.sh       |  68 ++
 .../netcdf-fortran@4.5.3.o33148097.exp-15-56       | 669 +++++++++++++++++++
 .../cpu/b/specs/nvhpc@21.9/netcdf-fortran@4.5.3.sh |  67 ++
 .../sdsc/expanse/0.17.3/cpu/b/yamls/compilers.yaml |  16 +
 18 files changed, 5213 insertions(+), 6 deletions(-)
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/cmake@3.21.4.o33147784.exp-15-56
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/cmake@3.21.4.sh
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/fftw@3.3.10-omp.o33150584.exp-15-56
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/mvapich2@2.3.7/fftw@3.3.10-omp.o33160030.exp-15-56
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/mvapich2@2.3.7/fftw@3.3.10-omp.sh
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9.o33146549.exp-15-56
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9.sh
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9/hdf5@1.10.7.o33147827.exp-15-56
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9/hdf5@1.10.7.sh
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9/netcdf-c@4.8.1.o33148039.exp-15-56
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9/netcdf-c@4.8.1.sh
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9/netcdf-fortran@4.5.3.o33148097.exp-15-56
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9/netcdf-fortran@4.5.3.sh
[mkandes@login01 spack]$ git log
commit a3029ae4c3bc806522bf34949c3e4ed995e6206b (HEAD -> sdsc-0.17.3, upstream/sdsc-0.17.3)
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Sat Aug 17 07:58:30 2024 -0700

    fftw@3.3.10 % gcc@10.2.0 +mpi +openmp failed builds
    
    Unfortauntely, I have not been able to resolve this issue yet. It's a
    bit odd that enabling +openmp causes the package build to fail when we
    already have successful builds with +mpi ~openmp. We will need to
    investigate this further in the future. For the time being, I will
    invetigate alternative compiler, MPI, and FFTW options for QE.

commit 5c15c7b17659e27a70d0dacf62449b589f65edca
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Sat Aug 17 07:19:07 2024 -0700

    Add fftw@3.3.10 % gcc@10.2.0 +openmp to exp/0.17.3/cpu/b
    
    Adding +openmp-enabled FFTW3 libraries to the production instances in an
    attempt to satisfy a requirement for the latest versions of Quantum
    Espresso, which appears to require a multithreaded FFTW-compatible
    libary.  For example, this is the build error I'm currently running into
[mkandes@login01 spack]$ git push
Username for 'https://github.com': ^C     
[mkandes@login01 spack]$ git push
Username for 'https://github.com': mkandes
Password for 'https://mkandes@github.com': 
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/mkandes/spack.git
   3cbfd1ba45..a3029ae4c3  sdsc-0.17.3 -> sdsc-0.17.3
[mkandes@login01 spack]$

[mkandes]

Create new feature branch for testing spec installation ...

[mkandes@login01 spack]$ git branch
* sdsc-0.17.3
  sdsc-0.17.3-gh-124-inst-exp-cpu-dev
  sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf
[mkandes@login01 spack]$ git checkout -b sdsc-0.17.3-gh-127-spec-exp-cpu-intel-tbb
Switched to a new branch 'sdsc-0.17.3-gh-127-spec-exp-cpu-intel-tbb'
[mkandes@login01 spack]$ git branch
  sdsc-0.17.3
  sdsc-0.17.3-gh-124-inst-exp-cpu-dev
  sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf
* sdsc-0.17.3-gh-127-spec-exp-cpu-intel-tbb
[mkandes@login01 spack]$

[mkandes]

Deploying intel-tbb@2020.3 to expanse/0.17.3/cpu/b ...

[spack_cpu@login01 ~]$ !7463
srun --partition=ind-shared --reservation=root_73  --account=use300 --nodes=1 --nodelist=exp-15-56 --ntasks-per-node=1 --cpus-per-task=16 --mem=32G --time=12:00:00 --pty --wait=0 /bin/bash
[spack_cpu@exp-15-56 ~]$ git log -1
fatal: not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[spack_cpu@exp-15-56 ~]$ cd /cm/shared/apps/spack/0.17.3/cpu/b
[spack_cpu@exp-15-56 b]$ git log -1
commit a3029ae4c3bc806522bf34949c3e4ed995e6206b (HEAD -> sdsc-0.17.3, origin/sdsc-0.17.3, origin/HEAD)
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Sat Aug 17 07:58:30 2024 -0700

    fftw@3.3.10 % gcc@10.2.0 +mpi +openmp failed builds
    
    Unfortauntely, I have not been able to resolve this issue yet. It's a
    bit odd that enabling +openmp causes the package build to fail when we
    already have successful builds with +mpi ~openmp. We will need to
    investigate this further in the future. For the time being, I will
    invetigate alternative compiler, MPI, and FFTW options for QE.
[spack_cpu@exp-15-56 b]$ git fetch origin
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (6/6), done.
Unpacking objects: 100% (16/16), 3.03 KiB | 119.00 KiB/s, done.
remote: Total 16 (delta 7), reused 15 (delta 7), pack-reused 0 (from 0)
From https://github.com/sdsc/spack
   a3029ae4c3..7db33edafe  sdsc-0.17.3 -> origin/sdsc-0.17.3
[spack_cpu@exp-15-56 b]$ git log -1
commit a3029ae4c3bc806522bf34949c3e4ed995e6206b (HEAD -> sdsc-0.17.3)
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Sat Aug 17 07:58:30 2024 -0700

    fftw@3.3.10 % gcc@10.2.0 +mpi +openmp failed builds
    
    Unfortauntely, I have not been able to resolve this issue yet. It's a
    bit odd that enabling +openmp causes the package build to fail when we
    already have successful builds with +mpi ~openmp. We will need to
    investigate this further in the future. For the time being, I will
    invetigate alternative compiler, MPI, and FFTW options for QE.
[spack_cpu@exp-15-56 b]$ git pull
Updating a3029ae4c3..7db33edafe
Fast-forward
 activate-shared-spack-instance.sh                  |  2 +-
 configure-shared-spack-instance.sh                 |  2 +-
 .../specs/intel@19.1.3.304/intel-mkl@2020.4.304.sh |  2 +-
 .../b/specs/intel@19.1.3.304/intel-tbb@2020.3.sh   | 73 ++++++++++++++++++++++
 4 files changed, 76 insertions(+), 3 deletions(-)
 mode change 100644 => 100755 configure-shared-spack-instance.sh
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/intel@19.1.3.304/intel-tbb@2020.3.sh
[spack_cpu@exp-15-56 b]$ cd etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/
[spack_cpu@exp-15-56 specs]$ cd intel@19.1.3.304/
[spack_cpu@exp-15-56 intel@19.1.3.304]$ sbatch intel-tbb@2020.3.sh 
sbatch: error: Batch job submission failed: Requested reservation is invalid
[spack_cpu@exp-15-56 intel@19.1.3.304]$ vi intel-tbb@2020.3.sh 
[spack_cpu@exp-15-56 intel@19.1.3.304]$ sbatch intel-tbb@2020.3.sh 
Submitted batch job 33452916
[spack_cpu@exp-15-56 intel@19.1.3.304]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          33452916 ind-share intel-tb spack_cp  R       0:04      1 exp-15-56
          33452904 ind-share     bash spack_cp  R       1:59      1 exp-15-56
[spack_cpu@exp-15-56 intel@19.1.3.304]$

[mkandes]

Deployed successfully ...

[mkandes@login02 ~]$ module load intel/19.1.3.304
[mkandes@login02 ~]$ module avail

 /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/intel/19.1.3.304 
   arpack-ng/3.8.0/ndrjsnd          netcdf-cxx4/4.3.1/d5kubqf
   cmake/3.21.4/bpzre3q             netcdf-fortran/4.5.3/hg7lhvi
   eigen/3.4.0/w2iz4rf              netlib-lapack/3.9.1/64ytaps
   fftw/3.3.10/jq4mbmk              openblas/0.3.18/sctoo2e-omp
   gsl/2.7/ix3kgnc                  openblas/0.3.18/xxq5a3j      (D)
   hdf5/1.10.7/butdr5c              openmpi/4.1.3/luhyajc
   hypre/2.23.0/w7eerxg             papi/6.0.0.1/tbdvjsm
   intel-mkl/2020.4.304/vg6aq26     petsc/3.16.1/ax5r6e3
   intel-mpi/2019.10.317/ezrfjne    sprng/5.0/u2op57v
   intel-tbb/2020.3/lfesfxm         superlu/5.3.0/tycq2th