Skip to content

SRE-3534 ci: adapt CI to use el9.7 #17376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
grom72 wants to merge 11 commits into
base: master
Choose a base branch
from
Draft

SRE-3534 ci: adapt CI to use el9.7 #17376

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
900350b
SRE-3522 ci: test CI with el9.7
grom72 Jan 14, 2026
8590393
Trigger FT skip NLT
grom72 Jan 15, 2026
1750cc8
Force 9.7 via Jenkins file
grom72 Jan 15, 2026
0cabf7b
Switch to EL9
grom72 Jan 15, 2026
eded0e1
Use DISTRO provided by pipeline-lib
grom72 Jan 15, 2026
afd3f01
Merge remote-tracking branch 'origin/master' into grom72/SRE-3522
grom72 Jan 16, 2026
fea5137
Trigger HW FT on el9.7
grom72 Jan 16, 2026
f02ea77
Trigger UT and FT on el9
grom72 Jan 16, 2026
a0e4556
Trigger HW FT only
grom72 Jan 16, 2026
3f8c7c2
Trigger only HW FT
grom72 Jan 16, 2026
6100e53
Trigger full build
grom72 Jan 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 29 additions & 28 deletions Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
/* groovylint-disable ParameterName, VariableName */
/* Copyright 2019-2024 Intel Corporation
/* Copyright 2025 Google LLC
* Copyright 2025 Hewlett Packard Enterprise Development LP
* Copyright 2025-2026 Hewlett Packard Enterprise Development LP
* All rights reserved.
*
* This file is part of the DAOS Project. It is subject to the license terms
Expand All @@ -19,6 +19,7 @@
// To use a test branch (i.e. PR) until it lands to master
// I.e. for testing library changes
//@Library(value='pipeline-lib@your_branch') _
@Library(value='pipeline-lib@grom72/SRE-3522') _
Copy link
Contributor

@daltonbohning daltonbohning Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to selves: we need to remove before landing


/* groovylint-disable-next-line CompileStatic */
job_status_internal = [:]
Expand Down Expand Up @@ -56,7 +57,7 @@ Map nlt_test() {
print 'Unstash failed, results from NLT stage will not be included'
}
sh label: 'Fault injection testing using NLT',
script: './ci/docker_nlt.sh --class-name el8.fault-injection fi'
script: './ci/docker_nlt.sh --class-name el9.fault-injection fi'
List filesList = []
filesList.addAll(findFiles(glob: '*.memcheck.xml'))
int vgfail = 0
Expand Down Expand Up @@ -282,7 +283,7 @@ pipeline {
description: 'Additional repository used for locating packages for the build and ' +
'test nodes, in the project@PR-number[:build] format.')
string(name: 'CI_HARDWARE_DISTRO',
defaultValue: '',
defaultValue: 'el9.7',
description: 'Distribution to use for CI Hardware Tests')
string(name: 'CI_EL8_TARGET',
defaultValue: '',
Expand Down Expand Up @@ -320,9 +321,9 @@ pipeline {
booleanParam(name: 'CI_FI_el8_TEST',
defaultValue: true,
description: 'Run the Fault injection testing on EL 8 test stage')
booleanParam(name: 'CI_TEST_EL8_RPMs',
booleanParam(name: 'CI_TEST_EL9_RPMs',
defaultValue: true,
description: 'Run the Test RPMs on EL 8 test stage')
description: 'Run the Test RPMs on EL 9 test stage')
booleanParam(name: 'CI_TEST_LEAP15_RPMs',
defaultValue: true,
description: 'Run the Test RPMs on Leap 15 test stage')
Expand Down Expand Up @@ -574,7 +575,7 @@ pipeline {
}
}
}
stage('Build on EL 9.6') {
stage('Build on EL 9.7') {
Copy link
Contributor

@JohnMalmberg JohnMalmberg Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For EL distros we generally intentionally build on el-X.Y older than current. Sometimes even numbered releases can have longer term support, and code built for el-9 of any older release in general should work on any older release.

when {
beforeAgent true
expression { !skip_build_stage('el9') }
Expand All @@ -590,7 +591,7 @@ pipeline {
' --build-arg DAOS_PACKAGES_BUILD=no ' +
' --build-arg DAOS_KEEP_SRC=yes ' +
' --build-arg REPOS="' + prRepos() + '"' +
' --build-arg POINT_RELEASE=.6 '
' --build-arg POINT_RELEASE=.7 '

}
}
Expand Down Expand Up @@ -726,7 +727,7 @@ pipeline {
expression { !skipStage() }
}
parallel {
stage('Unit Test on EL 8.8') {
stage('Unit Test on EL 9.7') {
Copy link
Contributor

@JohnMalmberg JohnMalmberg Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not be specifying the point release for tests, we should always test on the current point release.

We should also have the ability of a PR to test on a beta release of a point release. Currently our testing infrastructure is not setup to provide beta releases.

Copy link
Contributor

@daltonbohning daltonbohning Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just FYI that changing the names of these stages means the landing requirements need to be updated. I'll handle that when landing

when {
beforeAgent true
expression { !skipStage() }
Expand All @@ -748,7 +749,7 @@ pipeline {
}
}
}
stage('Unit Test bdev on EL 8.8') {
stage('Unit Test bdev on EL 9.7') {
when {
beforeAgent true
expression { !skipStage() }
Expand All @@ -770,7 +771,7 @@ pipeline {
}
}
}
stage('NLT on EL 8.8') {
stage('NLT on EL 9.7') {
when {
beforeAgent true
expression { params.CI_NLT_TEST && !skipStage() }
Expand All @@ -796,7 +797,7 @@ pipeline {
unitTestPost artifacts: ['nlt_logs/'],
testResults: 'nlt-junit.xml',
always_script: 'ci/unit/test_nlt_post.sh',
valgrind_stash: 'el8-gcc-nlt-memcheck'
valgrind_stash: 'el9-gcc-nlt-memcheck'
Copy link
Contributor

@JohnMalmberg JohnMalmberg Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pipeline-lib routine parseStageInfo should be providing the default value for the valgrind stash to make sure that it is correct and consistent instead of having to maintain this edit.

Copy link
Contributor

@daltonbohning daltonbohning Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO we should not rely on such a function to parse the stage name to determine the stash name. Functions of that nature have proven to be notorious to debug, modify, or even understand the flow.
It might be redundant here but I think this approach is actually better.

recordIssues enabledForFailure: true,
failOnError: false,
ignoreQualityGate: true,
Expand All @@ -810,7 +811,7 @@ pipeline {
}
}
}
stage('Unit Test with memcheck on EL 8.8') {
stage('Unit Test with memcheck on EL 9.7') {
when {
beforeAgent true
expression { !skipStage() }
Expand All @@ -830,12 +831,12 @@ pipeline {
always {
unitTestPost artifacts: ['unit_test_memcheck_logs.tar.gz',
'unit_test_memcheck_logs/**/*.log'],
valgrind_stash: 'el8-gcc-unit-memcheck'
valgrind_stash: 'el9-gcc-unit-memcheck'
job_status_update()
}
}
} // stage('Unit Test with memcheck on EL 8.8')
stage('Unit Test bdev with memcheck on EL 8.8') {
} // stage('Unit Test with memcheck on EL 9.7')
stage('Unit Test bdev with memcheck on EL 9.7') {
when {
beforeAgent true
expression { !skipStage() }
Expand All @@ -855,11 +856,11 @@ pipeline {
always {
unitTestPost artifacts: ['unit_test_memcheck_bdev_logs.tar.gz',
'unit_test_memcheck_bdev_logs/**/*.log'],
valgrind_stash: 'el8-gcc-unit-memcheck-bdev'
valgrind_stash: 'el9-gcc-unit-memcheck-bdev'
job_status_update()
}
}
} // stage('Unit Test bdev with memcheck on EL 8')
} // stage('Unit Test bdev with memcheck on EL 9.7')
}
}
stage('Test') {
Expand All @@ -870,7 +871,7 @@ pipeline {
expression { !paramsValue('CI_FUNCTIONAL_TEST_SKIP', false) }
}
parallel {
stage('Functional on EL 8.8 with Valgrind') {
stage('Functional on EL 9.7 with Valgrind') {
when {
beforeAgent true
expression { !skipStage() }
Expand All @@ -891,7 +892,7 @@ pipeline {
job_status_update()
}
}
} // stage('Functional on EL 8.8 with Valgrind')
} // stage('Functional on EL 9.7 with Valgrind')
stage('Functional on EL 8.8') {
when {
beforeAgent true
Expand All @@ -914,7 +915,7 @@ pipeline {
}
}
} // stage('Functional on EL 8.8')
stage('Functional on EL 9') {
stage('Functional on EL 9.7') {
Copy link
Contributor

@JohnMalmberg JohnMalmberg Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We control what el-9 point release is used for outside of Jenkins for these tests. The only time we don't want to test the current version is if we need to fix a problem with using the current point release, and the usual problem is that we are waiting for a custom driver from the high speed network adapter vendor, and that only affects hardware tests.

Copy link
Contributor

@daltonbohning daltonbohning Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. I think this should stay as Functional on EL 9. It also means we won't have to change the landing requirements for this one

Copy link
Contributor Author

@grom72 grom72 Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention of this change is to clearly indicate which version of OS we actually fully support.
This stems from release requirements, e.g., for 2.8, support for el9.7, and it is not clear anywhere whether this is maintained.

The alternative solution is to keep the original name, but start using the distro parameter in the getFunctionalTestStage function in Jenkinsfile
https://github.com/daos-stack/pipeline-lib/blob/cb2788756bc1179cda14db3c21650b05e400b175/vars/getFunctionalTestStage.groovy#L20

Copy link
Contributor

@daltonbohning daltonbohning Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think it's better to specify the distro explicitly. Putting 9.7 in the name of the stage does not necessarily mean we are running on 9.7. It's just a name.
Also, variables like CI_FUNCTIONAL_el9_TEST do not have the point release in them so omitting the point release in the name keeps them consistent.

when {
beforeAgent true
expression { !skipStage() }
Expand Down Expand Up @@ -981,7 +982,7 @@ pipeline {
}
} // post
} // stage('Functional on Ubuntu 20.04')
stage('Fault injection testing on EL 8.8') {
stage('Fault injection testing on EL 9.7') {
when {
beforeAgent true
expression { !skipStage() }
Expand Down Expand Up @@ -1031,16 +1032,16 @@ pipeline {
stash name: 'fault-inject-valgrind',
includes: '*.memcheck.xml',
allowEmpty: true
archiveArtifacts artifacts: 'nlt_logs/el8.fault-injection/',
archiveArtifacts artifacts: 'nlt_logs/el9.fault-injection/',
allowEmptyArchive: true
job_status_update()
}
}
} // stage('Fault injection testing on EL 8.8')
stage('Test RPMs on EL 8.6') {
} // stage('Fault injection testing on EL 9.7')
stage('Test RPMs on EL 9.6') {
when {
beforeAgent true
expression { params.CI_TEST_EL8_RPMs && !skipStage() }
expression { params.CI_TEST_EL9_RPMs && !skipStage() }
}
agent {
label params.CI_UNIT_VM1_LABEL
Expand All @@ -1056,7 +1057,7 @@ pipeline {
rpm_test_post(env.STAGE_NAME, env.NODELIST)
}
}
} // stage('Test RPMs on EL 8.6')
} // stage('Test RPMs on EL 9.6')
stage('Test RPMs on Leap 15.5') {
when {
beforeAgent true
Expand Down Expand Up @@ -1246,8 +1247,8 @@ pipeline {
} // stages
post {
always {
valgrindReportPublish valgrind_stashes: ['el8-gcc-nlt-memcheck',
'el8-gcc-unit-memcheck',
valgrindReportPublish valgrind_stashes: ['el9-gcc-nlt-memcheck',
'el9-gcc-unit-memcheck',
'fault-inject-valgrind']
job_status_update('final_status')
jobStatusWrite(job_status_internal)
Expand Down
4 changes: 2 additions & 2 deletions ci/unit/required_packages.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@

set -eu

# No longer used but provided by pipeline-lib
# Provided by pipeline-lib
# distro="$1"
# quick_build="${2:-false}"

OPENMPI_VER=""
PY_MINOR_VER=""

export DISTRO="el8" # should also work for el9
export DISTRO="${1:?ERROR: Missing distro argument. Usage: $0 <distro>}"
pkgs="$(utils/rpms/package_version.sh argobots lib) \
boost-python3$PY_MINOR_VER-devel \
capstone \
Expand Down
Loading