ROCP_SDK: Enable multi-GPU functionality.#562
Merged
Treece-Burgess merged 1 commit intoicl-utk-edu:masterfrom Feb 13, 2026
Merged
ROCP_SDK: Enable multi-GPU functionality.#562Treece-Burgess merged 1 commit intoicl-utk-edu:masterfrom
Treece-Burgess merged 1 commit intoicl-utk-edu:masterfrom
Conversation
660779f to
e872b19
Compare
Treece-Burgess
approved these changes
Feb 13, 2026
Contributor
Treece-Burgess
left a comment
There was a problem hiding this comment.
I tested this PR on Odyssey at Oregon with ROCm 7.2.0 pre-release (and ROCm 7.1.1 as a sanity test) with the following configure:
./configure --prefix=$PWD/test-install --with-components="rocp_sdk" --with-debug=yes
The results were:
- PAPI build: ✅
- PAPI utilities*: ✅
rocp_sdkcomponent tests: ✅
* papi_component_avail, papi_native_avail, and papi_command_line
Note that this PR resolves the test two_eventsets.c returning 0 values which began in ROCm 7.2.0:
# two_eventsets.c with ROCm 7.1.1 (master branch)
==================== FIRST EVENTSET - DEVICE 1 ====================
--------------------- PAPI_read()
rocp_sdk:::SQ_BUSY_CYCLES:device=0: 0 (0.00)
rocp_sdk:::SQ_BUSY_CYCLES:device=1: 2433452209 (1.87)
rocp_sdk:::TCC_CYCLE:device=1: 236585347099 (4.30)
rocp_sdk:::SQ_WAVES:device=0: 0 (0.00)
rocp_sdk:::SQ_WAVES:device=1: 5 (5.00)
--------------------- PAPI_read()
rocp_sdk:::SQ_BUSY_CYCLES:device=0: 0 (0.00)
rocp_sdk:::SQ_BUSY_CYCLES:device=1: 5068995890 (1.95)
rocp_sdk:::TCC_CYCLE:device=1: 490618431337 (4.46)
rocp_sdk:::SQ_WAVES:device=0: 0 (0.00)
rocp_sdk:::SQ_WAVES:device=1: 10 (5.00)
.
.
.
# two_eventsets.c with ROCm 7.2.0 (master branch)
==================== FIRST EVENTSET - DEVICE 1 ====================
--------------------- PAPI_read()
rocp_sdk:::SQ_BUSY_CYCLES:device=0: 0 (0.00)
rocp_sdk:::SQ_BUSY_CYCLES:device=1: 0 (0.00)
rocp_sdk:::TCC_CYCLE:device=1: 0 (0.00)
rocp_sdk:::SQ_WAVES:device=0: 0 (0.00)
rocp_sdk:::SQ_WAVES:device=1: 0 (0.00)
--------------------- PAPI_read()
rocp_sdk:::SQ_BUSY_CYCLES:device=0: 0 (0.00)
rocp_sdk:::SQ_BUSY_CYCLES:device=1: 0 (0.00)
rocp_sdk:::TCC_CYCLE:device=1: 0 (0.00)
rocp_sdk:::SQ_WAVES:device=0: 0 (0.00)
rocp_sdk:::SQ_WAVES:device=1: 0 (0.00)
.
.
.
# two_eventsets.c with ROCm 7.2.0 (this branch)
==================== FIRST EVENTSET - DEVICE 1 ====================
--------------------- PAPI_read()
rocp_sdk:::SQ_BUSY_CYCLES:device=0: 0 (0.00)
rocp_sdk:::SQ_BUSY_CYCLES:device=1: 2529921993 (1.95)
rocp_sdk:::TCC_CYCLE:device=1: 245688656588 (4.47)
rocp_sdk:::SQ_WAVES:device=0: 0 (0.00)
rocp_sdk:::SQ_WAVES:device=1: 5 (5.00)
--------------------- PAPI_read()
rocp_sdk:::SQ_BUSY_CYCLES:device=0: 0 (0.00)
rocp_sdk:::SQ_BUSY_CYCLES:device=1: 5294180824 (2.04)
rocp_sdk:::TCC_CYCLE:device=1: 512793339818 (4.66)
rocp_sdk:::SQ_WAVES:device=0: 0 (0.00)
rocp_sdk:::SQ_WAVES:device=1: 12 (6.00)
.
.
.
Rocm-7.2 encodes the device_id in the upper 32 bits of the counter id, so this comparison would fail for all GPUs except zero. Masking off the device_id by comparing only the lower 32 bits solves the problem.
e872b19 to
9e948bb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ROCm-7.2 encodes the device_id in the upper 32 bits of the counter id, so this comparison would fail for all GPUs except zero. Masking off the device_id by comparing only the lower 32 bits solves the problem.
Earlier versions of ROCm do not have this behavior, so the problem does not appear, and it will also be removed from future versions. This fix addresses a narrow range of versions for which this problem is present.
Pull Request Description
Author Checklist
Why this PR exists. Reference all relevant information, including background, issues, test failures, etc
Commits are self contained and only do one thing
Commits have a header of the form:
module: short descriptionCommits have a body (whenever relevant) containing a detailed description of the addressed problem and its solution
The PR needs to pass all the tests