SRE-3534 ci: adapt CI to use el9.7 #17376

grom72 · 2026-01-14T12:02:31Z

This PR upgrades the majority of the CI test stages so that they are run on EL9.7.

Requires:

SRE-3534 ci: update to el9.7 pipeline-lib#489

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Allow-unstable-test: true Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

github-actions · 2026-01-14T12:02:48Z

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/SRE-3534

Priority: 2 Skip-unit-tests:true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Cancel-prev-build: false Priority: 2 Allow-unstable-test: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

daosbuild3 · 2026-01-15T15:18:58Z

Test stage Unit Test bdev with memcheck on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/3/display/redirect

daosbuild3 · 2026-01-15T15:19:08Z

Test stage Unit Test on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/3/display/redirect

daosbuild3 · 2026-01-15T15:19:27Z

Test stage Unit Test bdev on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/3/display/redirect

daosbuild3 · 2026-01-15T15:19:57Z

Test stage Unit Test with memcheck on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/3/display/redirect

daosbuild3 · 2026-01-15T15:23:58Z

Test stage NLT on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/3/display/redirect

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

daosbuild3 · 2026-01-15T15:40:39Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17376/2/execution/node/738/log

daosbuild3 · 2026-01-15T15:40:41Z

Test stage Functional on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17376/2/execution/node/748/log

Cancel-prev-build: false Priority: 2 Skip-unit-tests:true Allow-unstable-test: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

daosbuild3 · 2026-01-15T17:29:57Z

Test stage Unit Test with memcheck on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/4/testReport/

daosbuild3 · 2026-01-15T18:19:02Z

Test stage NLT on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/4/testReport/

daosbuild3 · 2026-01-15T18:56:34Z

Test stage Unit Test bdev with memcheck on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/4/testReport/

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: false Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-func-vm: true Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Allow-unstable-test: true Skip-func-vm: true Skip-func-hw-test: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

daosbuild3 · 2026-01-16T08:06:41Z

Test stage Unit Test with memcheck on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/8/testReport/

daosbuild3 · 2026-01-16T08:47:57Z

Test stage NLT on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/8/testReport/

daosbuild3 · 2026-01-16T10:06:12Z

Test stage Unit Test bdev with memcheck on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/8/testReport/

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Skip-func-vm: true Skip-func-test-vm-valgrind: true Skip-func-test-el8: true Skip-func-test-el9: true Skip-func-test-leap15: true Skip-fault-injection-test: true Skip-test-el-9.6-rpms: false Skip-test-leap-15-rpms: true Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: false Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-func-vm: true Skip-func-test-vm-valgrind: true Skip-func-test-el8: true Skip-func-test-el9: true Skip-func-test-leap15: true Skip-fault-injection-test: true Skip-test-el-9.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: false Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

daosbuild3 · 2026-01-16T13:04:02Z

Test stage Functional on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/10/display/redirect

JohnMalmberg

Some general comments for potentially future cleanup.

JohnMalmberg · 2026-01-16T14:22:10Z

Jenkinsfile

                    }
                }
-                stage('Build on EL 9.6') {
+                stage('Build on EL 9.7') {


For EL distros we generally intentionally build on el-X.Y older than current. Sometimes even numbered releases can have longer term support, and code built for el-9 of any older release in general should work on any older release.

JohnMalmberg · 2026-01-16T14:24:54Z

Jenkinsfile

            }
            parallel {
-                stage('Unit Test on EL 8.8') {
+                stage('Unit Test on EL 9.7') {


We should not be specifying the point release for tests, we should always test on the current point release.

We should also have the ability of a PR to test on a beta release of a point release. Currently our testing infrastructure is not setup to provide beta releases.

Just FYI that changing the names of these stages means the landing requirements need to be updated. I'll handle that when landing

JohnMalmberg · 2026-01-16T14:27:47Z

Jenkinsfile

                                         testResults: 'nlt-junit.xml',
                                         always_script: 'ci/unit/test_nlt_post.sh',
-                                         valgrind_stash: 'el8-gcc-nlt-memcheck'
+                                         valgrind_stash: 'el9-gcc-nlt-memcheck'


The pipeline-lib routine parseStageInfo should be providing the default value for the valgrind stash to make sure that it is correct and consistent instead of having to maintain this edit.

IMO we should not rely on such a function to parse the stage name to determine the stash name. Functions of that nature have proven to be notorious to debug, modify, or even understand the flow.
It might be redundant here but I think this approach is actually better.

JohnMalmberg · 2026-01-16T14:32:17Z

Jenkinsfile

                    }
                } // stage('Functional on EL 8.8')
-                stage('Functional on EL 9') {
+                stage('Functional on EL 9.7') {


We control what el-9 point release is used for outside of Jenkins for these tests. The only time we don't want to test the current version is if we need to fix a problem with using the current point release, and the usual problem is that we are waiting for a custom driver from the high speed network adapter vendor, and that only affects hardware tests.

+1. I think this should stay as Functional on EL 9. It also means we won't have to change the landing requirements for this one

The intention of this change is to clearly indicate which version of OS we actually fully support.
This stems from release requirements, e.g., for 2.8, support for el9.7, and it is not clear anywhere whether this is maintained.

The alternative solution is to keep the original name, but start using the distro parameter in the getFunctionalTestStage function in Jenkinsfile
https://github.com/daos-stack/pipeline-lib/blob/cb2788756bc1179cda14db3c21650b05e400b175/vars/getFunctionalTestStage.groovy#L20

I do think it's better to specify the distro explicitly. Putting 9.7 in the name of the stage does not necessarily mean we are running on 9.7. It's just a name.
Also, variables like CI_FUNCTIONAL_el9_TEST do not have the point release in them so omitting the point release in the name keeps them consistent.

daosbuild3 · 2026-01-16T14:59:33Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17376/11/execution/node/756/log

daosbuild3 · 2026-01-16T16:20:37Z

Test stage Functional Hardware Large completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17376/11/execution/node/674/log

ci/unit/required_packages.sh

daltonbohning · 2026-01-16T17:44:30Z

Jenkinsfile

 // To use a test branch (i.e. PR) until it lands to master
 // I.e. for testing library changes
 //@Library(value='pipeline-lib@your_branch') _
+@Library(value='pipeline-lib@grom72/SRE-3522') _


Note to selves: we need to remove before landing

daltonbohning · 2026-01-16T17:46:09Z

Jenkinsfile

            }
            parallel {
-                stage('Unit Test on EL 8.8') {
+                stage('Unit Test on EL 9.7') {


Just FYI that changing the names of these stages means the landing requirements need to be updated. I'll handle that when landing

daltonbohning · 2026-01-16T17:47:54Z

Jenkinsfile

                                         testResults: 'nlt-junit.xml',
                                         always_script: 'ci/unit/test_nlt_post.sh',
-                                         valgrind_stash: 'el8-gcc-nlt-memcheck'
+                                         valgrind_stash: 'el9-gcc-nlt-memcheck'


IMO we should not rely on such a function to parse the stage name to determine the stash name. Functions of that nature have proven to be notorious to debug, modify, or even understand the flow.
It might be redundant here but I think this approach is actually better.

daltonbohning · 2026-01-16T17:49:47Z

Jenkinsfile

                    }
                } // stage('Functional on EL 8.8')
-                stage('Functional on EL 9') {
+                stage('Functional on EL 9.7') {


+1. I think this should stay as Functional on EL 9. It also means we won't have to change the landing requirements for this one

daosbuild3 · 2026-01-16T18:52:24Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17376/11/execution/node/922/log

daosbuild3 · 2026-01-21T11:20:57Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17376/16/execution/node/818/log

daosbuild3 · 2026-01-21T11:34:31Z

Test stage Functional Hardware Large completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17376/16/execution/node/767/log

daosbuild3 · 2026-01-21T11:36:34Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17376/16/execution/node/757/log

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Cancel-prev-build: false Priority: 2 Allow-unstable-test: true Skip-func-test-vm-valgrind: false Skip-func-test-el8: false Skip-func-test-el9: false Skip-func-hw-test-medium: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false

daosbuild3 · 2026-01-21T20:47:14Z

Test stage NLT on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/19/testReport/

daosbuild3 · 2026-01-21T20:54:59Z

Test stage Unit Test with memcheck on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/19/testReport/

daosbuild3 · 2026-01-21T22:41:55Z

Test stage Unit Test bdev with memcheck on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/19/testReport/

grom72 added 2 commits January 15, 2026 15:38

Trigger FT skip NLT

8590393

Priority: 2 Skip-unit-tests:true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Force 9.7 via Jenkins file

1750cc8

Cancel-prev-build: false Priority: 2 Allow-unstable-test: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Switch to EL9

0cabf7b

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Use DISTRO provided by pipeline-lib

eded0e1

Cancel-prev-build: false Priority: 2 Skip-unit-tests:true Allow-unstable-test: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 added 3 commits January 16, 2026 07:49

Merge remote-tracking branch 'origin/master' into grom72/SRE-3522

afd3f01

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Trigger UT and FT on el9

f02ea77

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Allow-unstable-test: true Skip-func-vm: true Skip-func-hw-test: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 mentioned this pull request Jan 16, 2026

SRE-3534 ci: update to el9.7 daos-stack/pipeline-lib#489

Open

grom72 requested review from JohnMalmberg, daltonbohning, phender and ryon-jensen January 16, 2026 12:50

grom72 changed the title ~~SRE-3522 ci: test CI with el9.7~~ SRE-3534 ci: test CI with el9.7 Jan 16, 2026

JohnMalmberg approved these changes Jan 16, 2026

View reviewed changes

grom72 changed the title ~~SRE-3534 ci: test CI with el9.7~~ SRE-3534 ci: CI to use el9.7 (quick solution) Jan 16, 2026

daltonbohning reviewed Jan 16, 2026

View reviewed changes

grom72 changed the title ~~SRE-3534 ci: CI to use el9.7 (quick solution)~~ SRE-3534 ci: adapt CI to use el9.7 Jan 19, 2026

SRE-3534 ci: adapt CI to use el9.7 #17376

Are you sure you want to change the base?

SRE-3534 ci: adapt CI to use el9.7 #17376

Uh oh!

Conversation

grom72 commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions bot commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Jan 15, 2026

Uh oh!

daosbuild3 commented Jan 15, 2026

Uh oh!

daosbuild3 commented Jan 15, 2026

Uh oh!

daosbuild3 commented Jan 15, 2026

Uh oh!

daosbuild3 commented Jan 15, 2026

Uh oh!

daosbuild3 commented Jan 15, 2026

Uh oh!

daosbuild3 commented Jan 15, 2026

Uh oh!

daosbuild3 commented Jan 15, 2026

Uh oh!

daosbuild3 commented Jan 15, 2026

Uh oh!

daosbuild3 commented Jan 15, 2026

Uh oh!

daosbuild3 commented Jan 16, 2026

Uh oh!

daosbuild3 commented Jan 16, 2026

Uh oh!

daosbuild3 commented Jan 16, 2026

Uh oh!

daosbuild3 commented Jan 16, 2026

Uh oh!

JohnMalmberg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Jan 16, 2026

Uh oh!

daosbuild3 commented Jan 16, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Jan 16, 2026

Uh oh!

daosbuild3 commented Jan 21, 2026

Uh oh!

daosbuild3 commented Jan 21, 2026

grom72 commented Jan 14, 2026 •

edited

Loading

github-actions bot commented Jan 14, 2026 •

edited

Loading