fix(rocprofiler-sdk): stabilize gfx12 CI and profiling tests by venkat1361 · Pull Request #11732 · ROCm/rocm-systems

venkat1361 · 2026-09-16T21:41:25Z

Motivation

CI runs on Navi4/gfx12 exposed failures caused by zero counter values in AUTO performance mode, interference between concurrent GPU tests, unsupported or hidden GPU agents, and timestamp adjustments that could place dispatch completion after correlation-ID retirement.

This PR moves the SLES 15.6 job to a Navi4 runner and addresses these failures while preserving strict validation of available counters and replay structure.

Technical Details

Run SLES 15.6 CI on gfx12.
Handle zero counters in AUTO mode. Query GPU 0’s performance level only for gfx11* and gfx12*. Skip the affected Perfetto counter checks when all collected counters are zero.
Restrict the buffered counter sample to GPUs visible through both HSA and HIP
Preserve measured duration when an interval fits inside the CPU window; otherwise, clamp both endpoints. This prevents correction from moving completion beyond the CPU completion handler and correlation retirement.
Adopt the shared integration execute/validate helpers for kernel replay, replay performance, and kernel-trace duration tests. Execution inherits serial scheduling, and performance validation runs through pytest with its existing limits.
Disable unstable HIP graph-bubbles tests on gfx1200 and gfx1201, extending the existing gfx1151 exclusion.

Issue Tracking

Issue ID: #10876

Test Plan

Existing test and new test.

Test Result

Pass

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

therock-pr-bot · 2026-09-16T21:42:02Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-09-16T21:42:03Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

Copilot

🔵 Needs a closer look

It combines production timestamp behavior changes with extensive GPU-specific test and CI restructuring that requires hardware validation.

Pull request overview

Improves rocprofiler-sdk reliability and test coverage on gfx120X systems.

Changes:

Handles zero counters under AUTO GPU performance mode.
Migrates replay tests to standardized integration helpers and adds gfx120X CI.
Corrects profiling timestamps and skips unsupported GPU configurations.

File summaries

File	Description
`tracing-plus-counter-collection/validate.py`	Allows zero AUTO-mode counters.
`tracing-plus-counter-collection/conftest.py`	Detects AUTO performance mode.
`kernel-trace-duration/CMakeLists.txt`	Migrates integration test registration.
`kernel-replay/validate.py`	Separates and conditionally skips counter checks.
`kernel-replay/validate_perf.py`	Converts validation to pytest.
`kernel-replay/conftest.py`	Adds performance options and parametrization.
`kernel-replay/CMakeLists.txt`	Migrates replay integration tests.
`hip-graph-bubbles-test/CMakeLists.txt`	Disables unsupported gfx120X tests.
`pytest_utils/__init__.py`	Adds GPU performance-level detection.
`kernel-replay-perf/CMakeLists.txt`	Serializes performance integration tests.
`profiling_time.hpp`	Clamps skewed timestamps while preserving duration.
`timestamp.cpp`	Tests profiling-time bounds.
`metrics_test.cpp`	Handles GPUs without counter support.
`kernel_replay_performance.md`	Updates generated test name.
`counter_collection/CMakeLists.txt`	Serializes the functional counter test.
`buffered_client.cpp`	Filters GPUs by runtime visibility.
`rocprofiler-sdk-continuous_integration.yml`	Adds a gfx120X CI runner.

Review details

Files reviewed: 17/17 changed files
Comments generated: 0
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

ApoKalipse-V

Approved except for the clock part. That I'd like the owner of that code to review.

github-actions · 2026-10-14T19:49:34Z

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

…uto.

jbonnell-amd

Approved for the .github/** changes

ApoKalipse-V

Re-approve

arvindcheru · 2026-10-19T14:47:18Z

Override Multi-Arch CI Summary only, at f7def57: rocm-libraries#13766, rocm-libraries#13414

github-actions Bot added the github actions Pull requests that update GitHub Actions code label Sep 16, 2026

therock-pr-bot Bot added the Not ready to Review label Sep 16, 2026

systems-assistant Bot added the organization: ROCm label Sep 16, 2026

github-actions Bot added project: rocprofiler-sdk documentation Improvements or additions to documentation labels Sep 16, 2026

venkat1361 linked an issue Sep 17, 2026 that may be closed by this pull request

[Issue] Linux : rocprofiler-sdk fails 5 tests — counter_collection empty, thread-trace triple-buffer timeouts, kernel-trace bubble threshold #10876

Closed

venkat1361 requested a balanced review from Copilot September 17, 2026 16:53

Copilot started reviewing on behalf of venkat1361 September 17, 2026 16:54

View session

Copilot AI reviewed Sep 17, 2026

View reviewed changes

venkat1361 changed the title ~~fix(rocprofiler-sdk): fix att and counter collection tests on gfx120X~~ fix(rocprofiler-sdk): stabilize gfx12 CI and profiling tests Sep 17, 2026

venkat1361 marked this pull request as ready for review September 17, 2026 17:22

venkat1361 requested review from a team as code owners September 17, 2026 17:22

venkat1361 removed the Not ready to Review label Sep 17, 2026

ApoKalipse-V reviewed Sep 17, 2026

View reviewed changes

ApoKalipse-V approved these changes Sep 17, 2026

View reviewed changes

jrmadsen reviewed Sep 22, 2026

View reviewed changes

venkat1361 requested a review from jrmadsen September 22, 2026 19:47

venkat1361 force-pushed the users/venkat1361/gfx12-rocprofiler-tests-fix branch from 0732ad0 to 3667806 Compare September 23, 2026 20:02

amd-chiranjeevi mentioned this pull request Sep 25, 2026

[Issue] Linux : rocprofiler-sdk fails 5 tests — counter_collection empty, thread-trace triple-buffer timeouts, kernel-trace bubble threshold #10876

Closed

venkat1361 force-pushed the users/venkat1361/gfx12-rocprofiler-tests-fix branch from 3667806 to 156d998 Compare September 29, 2026 18:26

venkat1361 mentioned this pull request Oct 1, 2026

fix(libhsakmt): reduce GPU/CPU clock-correlation errors #12665

Closed

1 task

venkat1361 force-pushed the users/venkat1361/gfx12-rocprofiler-tests-fix branch 2 times, most recently from 8ed25c3 to d279f12 Compare October 14, 2026 18:36

venkat1361 added 2 commits October 15, 2026 11:28

move sles check to navi4 runner.

7859ee9

skip counter collection test when on gfx11/12 with perf mode set to a…

27550fe

…uto.

venkat1361 added 10 commits October 15, 2026 11:28

misc. fixes.

8f5abc8

fix more tests.

3877830

refactor to use rocprofiler_add_integration_ for integration tests.

ca7282e

format.

a8f65e3

address comments.

01f156a

use for loop to disable bubbles tests in specific gfx archs.

f05537c

make chagnes minimal in clock shifting.

aa50c31

revert changes in timestamps.

32bc644

fix(rocprofiler-sdk): preserve upstream replay test after rebase

c0b65b3

add AMD GPU driver version in CI step.

530cee6

venkat1361 force-pushed the users/venkat1361/gfx12-rocprofiler-tests-fix branch from d279f12 to 530cee6 Compare October 15, 2026 16:28

venkat1361 mentioned this pull request Oct 15, 2026

[Issue]: [rocprofiler-sdk][gfx120x] Kernel-replay SPM tests and sample abort during KFD acquisition in CI #13606

Open

venkat1361 force-pushed the users/venkat1361/gfx12-rocprofiler-tests-fix branch from ce3f4e7 to ab9e9b3 Compare October 15, 2026 22:50

jbonnell-amd approved these changes Oct 16, 2026

View reviewed changes

venkat1361 added 5 commits October 16, 2026 17:54

Set barrier-wait exclusion unconditionally in GFX12 occupancy tracing

3152114

disable spm tests on gfx120X CI check.

1bedd30

misc.

09c3bac

Navi 4 gpu workarond ci.

67fcd90

Keep Git's global config writable even when /github/home is absent.

0b3dc99

venkat1361 force-pushed the users/venkat1361/gfx12-rocprofiler-tests-fix branch from f9d124c to 0b3dc99 Compare October 16, 2026 17:55

ApoKalipse-V approved these changes Oct 16, 2026

View reviewed changes

add gfx1200 to GPU_TARGETS.

f7def57

arvindcheru merged commit f94e399 into develop Oct 19, 2026
851 of 881 checks passed

arvindcheru deleted the users/venkat1361/gfx12-rocprofiler-tests-fix branch October 19, 2026 14:47

Conversation

venkat1361 commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Issue Tracking

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Policy Checks Passed

Uh oh!

therock-pr-bot Bot commented Sep 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ApoKalipse-V left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026

Code Coverage Report

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

Uh oh!

jbonnell-amd left a comment

Choose a reason for hiding this comment

Uh oh!

ApoKalipse-V left a comment

Choose a reason for hiding this comment

Uh oh!

arvindcheru commented Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

venkat1361 commented Sep 16, 2026 •

edited

Loading

therock-pr-bot Bot commented Sep 16, 2026 •

edited

Loading