Spack Working Group @ SDSC - Meeting - 2023/03/24

[mkandes]

Attendance

• Marty Kandes
• Mahidahr Tatineni
• Subhashini Sivagnanam
• Jerry Greenberg
• Nicole Wolter
• Yuwu Chen
• Trevor Cooper

Agenda

1. No Meeting Next Week

Friday, March 31st is Cesar Chavez Day (Holiday)

2. Upcoming Planned Maintenance on Expanse

Expected date of maintenance: April 2023 - May 2023

Key changes expected to system software:

• Update operating system from Rocky 8.5 to Rocky 8.7
• Update Infiniband driver from Mellanox OFED 4.9-4.1.7 to Mellanox OFED 4.9-6.0.6 Mellanox OFED 5.8-1.1.2
• Update Lustre client from lfs 2.12.8_6_g5457c37 to lfs 2.15.2
• Update NVIDIA GPU driver from NVIDIA 510.39.01 CUDA 11.6 to NVIDIA 515.65.01 CUDA 11.7

Plan to deploy new tests instances on Expanse development system (devexp) this weekend

3. Update on pre-production testing of expanse/0.17.3/cpu/a and expanse/0.17.3/gpu/a

4. Planning and Preparation for Deployment of Spack on TSCC2

Expected date of deployment: April 2023
Expected operating system at deployment: Rocky 9.x; base gcc moves to 11.2.x

Update from Jerry on status of tscc2/0.17.3/cpu and tscc2/0.17.3/gpu instance build out and testing

• https://github.com/jerrygreenberg2/sdsc

5. Other things to do

• Investigate creating a build cache on Jetstream2; may allow us to deploy more quickly during system maintenance
• Complete initial draft of DEPLOYMENT.md, which will serve as general deployment documentation
• Complete initial draft of CONTRIBUTING.md, which will serve as documentation on how to contribute work to this repo
• Create a dev & test spack instance(s) based on a subset of the the expanse/0.17.3/cpu and expanse/0.17.3/gpu instances
• Fix ordering of builds in dependency chains; lmod appears to set the last installed variant of a package as the default

References:

• https://forge.sdsc.edu/confluence/display/HO/Expanse+Acceptance+Testing
• https://forge.sdsc.edu/confluence/display/HO/Expanse+Spack+Application+Deployment

[mkandes]

@mkandes - Roll new deployment branch for Spack v0.19.1

[mkandes]

Done.

[mkandes]

@mkandes - rdma-core version for MLNX_OFED 5.8-1.1.2

[mkandes@devexp-2-01 ~]$ srun --partition=shared --account=use300 --ntasks-per-node=10 --nodes=1 --mem=32G --time=00:30:00 --wait=0 --pty /bin/bash
[mkandes@devexp-1-01 ~]$ ls -lahtr /usr/lib64/libibverbs.so.1
lrwxrwxrwx 1 root root 23 Nov 27 02:30 /usr/lib64/libibverbs.so.1 -> libibverbs.so.1.14.43.0
[mkandes@devexp-1-01 ~]$ exit
exit
[mkandes@devexp-2-01 ~]$ srun --partition=gpu-shared --account=use300 --ntasks-per-node=10 --nodes=1 --mem=93G --gpus=1 --time=00:30:00 --wait=0 --pty /bin/bash
[mkandes@devexp-3-01 ~]$ ls -lahtr /usr/lib64/libibverbs.so.1
lrwxrwxrwx 1 root root 23 Nov 27 02:30 /usr/lib64/libibverbs.so.1 -> libibverbs.so.1.14.43.0
[mkandes@devexp-3-01 ~]$ exit
exit
[mkandes@devexp-2-01 ~]$ ls -lahtr /usr/lib64/libibverbs.so.1
lrwxrwxrwx 1 root root 23 Nov 27 02:30 /usr/lib64/libibverbs.so.1 -> libibverbs.so.1.14.43.0
[mkandes@devexp-2-01 ~]$