Skip to content

Add new GPU workflow #949

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: 2023.06-software.eessi.io
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .github/workflows/parse_missing-installations-output.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import os
import re

missing = os.environ['missing']
missing = missing.split('\n')
missing_cuda = []
missing_cpu = []
for ec in missing:
if re.search('CUDA', ec):
missing_cuda.append(ec)
else:
missing_cpu.append(ec)
if len(missing_cpu) != 0 and len(missing_cuda) != 0:
print(f'Please open a seperate pr for these dependencies: {missing_cpu}')
os.write(2, b'Error: CPU dependencies for CUDA build must be build in a seperate pr')
exit(1)
elif len(missing_cuda) != 0:
# TODO: Make this set the accelorator label?
print(f'Have fun installing the following gpu builds: {missing_cuda}')
elif len(missing_cpu) != 0:
print(f'Have fun installing the following gpu builds: {missing_cpu}')
else:
print('no missing modules')
77 changes: 77 additions & 0 deletions .github/workflows/test-check_easystacks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# documentation: https://help.github.com/en/articles/workflow-syntax-for-github-actions
name: Check for CUDA builds in changed easystacks
on:
push:
branches: [ "*-software.eessi.io" ]
pull_request:
workflow_dispatch:
permissions:
contents: read # to fetch code (actions/checkout)
env:
EESSI_ACCELERATOR_TARGETS: |
x86_64/amd/zen2:
- nvidia/cc80
x86_64/amd/zen3:
- nvidia/cc80
jobs:
check_missing:
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
EESSI_VERSION:
- 2023.06
EESSI_SOFTWARE_SUBDIR_OVERRIDE:
- x86_64/amd/zen2
- x86_64/amd/zen3
- x86_64/amd/zen4
- x86_64/intel/haswell
- x86_64/intel/skylake_avx512
- x86_64/generic
steps:
- name: Check out software-layer repository
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

- name: Mount EESSI CernVM-FS pilot repository
uses: cvmfs-contrib/github-action-cvmfs@55899ca74cf78ab874bdf47f5a804e47c198743c # v4.0
with:
cvmfs_config_package: https://github.com/EESSI/filesystem-layer/releases/download/latest/cvmfs-config-eessi_latest_all.deb
cvmfs_http_proxy: DIRECT
cvmfs_repositories: software.eessi.io
- name: find changed easystacks
uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 #v3.0.2
id: changes
with:
list-files: shell
filters: |
changed:
- 'easystacks/**'
- name: Test check_missing_installations.sh script
if: ${{ steps.changes.outputs.changed == 'true' }}
run: |
export EESSI_SOFTWARE_SUBDIR_OVERRIDE=${{matrix.EESSI_SOFTWARE_SUBDIR_OVERRIDE}}
source /cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}/init/bash
# set $EESSI_CPU_FAMILY to the CPU architecture that corresponds to $EESSI_SOFTWARE_SUBDIR_OVERRIDE (part before the first slash),
# to prevent issues with checks in the Easybuild configuration that use this variable
export EESSI_CPU_FAMILY=${EESSI_SOFTWARE_SUBDIR_OVERRIDE%%/*}
module load EasyBuild
which eb
eb --version
export EESSI_PREFIX=/cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}
export EESSI_OS_TYPE=linux
env | grep ^EESSI | sort

# check for missing installations if a file in the easystacks directory is changed
echo "check for missing installation in changed easystack files"
easystack_files=(${{ steps.changes.outputs.changed_files }})

for easystack_file in ${easystack_files[@]}; do
echo "check missing installatios for ${easystack_file}..."
./check_missing_installations.sh ${easystack_file} || ec=$?
export eb_missing_out=$(./check_missing_installations.sh ${easystack_file})
if [[ ${ec} -ne 0 ]]; then
echo "missing installations found for ${easystack_file}!" >&2
export missing=$(env | grep .eb\) | tr -d \*)
python .github/workflows/parse_missing-installations-output.py
fi
done
6 changes: 2 additions & 4 deletions .github/workflows/test-software.eessi.io.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,7 @@ jobs:
echo "just run check_missing_installations.sh (should use easystacks/software.eessi.io/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-*.yml with latest EasyBuild release)"
for easystack_file in $(EESSI_VERSION=${{matrix.EESSI_VERSION}} .github/workflows/scripts/only_latest_easystacks.sh); do
echo "check missing installations for ${easystack_file}..."
./check_missing_installations.sh ${easystack_file}
ec=$?
./check_missing_installations.sh ${easystack_file} || ec=$?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this will work, it only sets ec if the script did not exit successfully

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that is what I want otherwise the workflow stops. And than this error message does not get printed "missing installations found for ${easystack_file}!"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also tested it with an easystack were everything was already installed and that went fine as well.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can test again to see if I overlooked something.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah no your right

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok than I'll have to find another way around it

if [[ ${ec} -ne 0 ]]; then echo "missing installations found for ${easystack_file}!" >&2; exit ${ec}; fi
done

Expand All @@ -75,8 +74,7 @@ jobs:
echo "checking missing installations for accelerator ${accel} using modulepath: ${MODULEPATH}"
for easystack_file in $(EESSI_VERSION=${{matrix.EESSI_VERSION}} ACCEL_EASYSTACKS=1 .github/workflows/scripts/only_latest_easystacks.sh); do
echo "check missing installations for ${easystack_file}..."
./check_missing_installations.sh ${easystack_file}
ec=$?
./check_missing_installations.sh ${easystack_file} || ec=$?
if [[ ${ec} -ne 0 ]]; then echo "missing installations found for ${easystack_file}!" >&2; exit ${ec}; fi
done
module unuse ${EESSI_SOFTWARE_PATH}/accel/${accel}/modules/all
Expand Down