Skip to content

SRE-3704 ci: CI-Test-FI#17953

Draft
grom72 wants to merge 48 commits intomasterfrom
grom72/SRE-3704-CI-Test-FI
Draft

SRE-3704 ci: CI-Test-FI#17953
grom72 wants to merge 48 commits intomasterfrom
grom72/SRE-3704-CI-Test-FI

Conversation

@grom72
Copy link
Copy Markdown
Contributor

@grom72 grom72 commented Apr 9, 2026

CI Tests of FI stage.

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 9, 2026

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/SRE-3704

@grom72 grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 4 times, most recently from 276641f to 23827b4 Compare April 13, 2026 07:37
Comment thread utils/docker/Dockerfile.el.9 Fixed
Comment thread utils/docker/Dockerfile.el.9 Fixed
@grom72 grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 3 times, most recently from e724b71 to 14b4ae9 Compare April 16, 2026 13:25
@daosbuild3
Copy link
Copy Markdown
Collaborator

@daosbuild3
Copy link
Copy Markdown
Collaborator

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17953/56/display/redirect

1 similar comment
@daosbuild3
Copy link
Copy Markdown
Collaborator

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17953/56/display/redirect

@grom72 grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 2 times, most recently from eea6d40 to 8bf0a15 Compare April 20, 2026 07:48
@daosbuild3
Copy link
Copy Markdown
Collaborator

@grom72 grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 3 times, most recently from 3591870 to fdc56a7 Compare April 20, 2026 14:09
@daosbuild3
Copy link
Copy Markdown
Collaborator

@grom72 grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 3 times, most recently from dd13687 to 07365a4 Compare April 21, 2026 13:00
@daosbuild3
Copy link
Copy Markdown
Collaborator

@grom72 grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch from 45aa283 to 51c7487 Compare April 29, 2026 10:44
grom72 added 6 commits April 29, 2026 15:27
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Priority: 2

Skip-python-bandit: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: true
Skip-unit-test-memcheck: true

Skip-func-vm: true
Skip-test-el-9.6-rpms: true
Skip-test-leap-15-rpms: true

Skip-func-hw-test: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Skip-python-bandit: true

Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: true
Skip-unit-test-memcheck: true

Skip-func-test-el8: true
Skip-func-test-el9: true
Skip-func-test-leap15: true
Skip-test-el-9.6-rpms: true
Skip-test-leap-15-rpms: true

Skip-func-hw-test: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Cancel-prev-build: false

Skip-python-bandit: true

Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: true
Skip-unit-test-memcheck: true

Skip-func-test-el9: true
Skip-test-el-9.6-rpms: true
Skip-test-leap-15-rpms: true

Skip-func-hw-test: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Cancel-prev-build: false

Skip-python-bandit: true

Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: true
Skip-unit-test-memcheck: true

Skip-func-test-el9: true
Skip-test-el-9.6-rpms: true
Skip-test-leap-15-rpms: true

Skip-func-hw-test: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Cancel-prev-build: false
Priority: 2
grom72 added 2 commits April 29, 2026 15:28
unitTestPost() already processes nlt-junit.xml via the testResults
parameter it receives. The bare 'junit testResults: nlt-junit.xml'
call that follows is redundant and has no failure protection: it uses
the default healthScaleFactor so when fault injection tests
intentionally produce failures in nlt-junit.xml it marks the build
FAILURE immediately, overriding the controlled result handling done
by unitTestPost().

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Pass nlt_name: 'Fault injection issues' to unitTestPost() in the
NLT Fault injection testing post block so that the recordIssues
warnings section is displayed as 'Fault injection issues' in Jenkins
rather than the generic 'Node local testing' used by the NLT stage.

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
@grom72 grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch from 51c7487 to 94807ff Compare April 29, 2026 13:52
When node_local_test.py runs with --no-root, DAOS logs are written to
/localhome/jenkins/build/nlt_logs/ instead of /tmp/. The existing rsync
only fetches from /tmp/, leaving nlt_logs/ empty and causing:

  No artifacts found that match the file pattern "nlt_logs/". Configuration error?

Add a second rsync from build/nlt_logs/ to collect logs from the --no-root
code path. The '|| true' ensures non-fatal behavior when the path does not
exist (plain NLT runs without --no-root).

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
@grom72 grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch from 94807ff to df82b57 Compare April 29, 2026 13:54
grom72 added 4 commits April 29, 2026 17:31
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
…Test-FI

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
The vm_test/nlt-errors.json issue scanning for the 'NLT Fault injection
testing' stage is now handled by unitTestPost() in pipeline-lib, so
remove it from the explicit recordIssues call here.

Switch from tools: (list) to tool: (single) and keep only
nlt-client-leaks.json, renaming its id to NLT_client_from_J to avoid
collision with the id used by pipeline-lib.

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
@daosbuild3
Copy link
Copy Markdown
Collaborator

@daosbuild3
Copy link
Copy Markdown
Collaborator

@daosbuild3
Copy link
Copy Markdown
Collaborator

@daosbuild3
Copy link
Copy Markdown
Collaborator

Run NLT and Fault Injection Tests no dedicated VMs with 64GiB of memory
reserved.

Limit NLT memory to 16GiB

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
@daosbuild3
Copy link
Copy Markdown
Collaborator

@daosbuild3
Copy link
Copy Markdown
Collaborator

Test stage NLT Fault injection testing completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17953/92/display/redirect

grom72 added 13 commits May 5, 2026 01:47
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2

Cancel-prev-build: false

Skip-python-bandit: true

Skip-unit-test: true

Skip-unit-test-memcheck: true

Skip-func-vm-all: true

Skip-test-el-9-rpms: true

Skip-test-leap-15-rpms: true

Skip-func-hw-test: true

Skip-build-el8-gcc: true

Skip-build-leap15-gcc: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Skip-func-test-el9: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Skip-func-test-el9: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Skip-func-test-el9: true
This reverts commit 2323fd9.
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Skip-func-test-el9: true
Fault injection must have NLT in stage name
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Skip-func-test-el9: true
This reverts commit b03decb.

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Skip-func-test-el9: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Skip-func-test-el9: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Priority: 2
Cancel-prev-build: false
Skip-python-bandit: true
Skip-unit-test: true
Skip-unit-test-memcheck: true
Skip-func-vm-all: true
Skip-test-el-9-rpms: true
Skip-test-leap-15-rpms: true
Skip-func-hw-test: true
Skip-build-el8-gcc: true
Skip-build-leap15-gcc: true
Skip-func-test-el9: true
@grom72 grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch from d1a18e4 to bdd0209 Compare May 6, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants