feat(rccl): Add LL128 support for DDA RS by nusislam · Pull Request #12928 · ROCm/rocm-systems

nusislam · 2026-10-06T17:06:53Z

Motivation

This PR adds LL128 protocol support for DDA RS in the fabric path.

Technical Details

This PR adds LL128 protocol support for DDA RS in the fabric path. It also moves the copies to temporary buffer inside the kernel for some message sizes in the medium range [internal-env].

Issue Tracking

AICOMRCCL-2471

Test Plan

rccl-test, unit tests

Test Result

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

github-actions · 2026-10-06T17:07:46Z

No undocumented RCCL or NCCL environment-variable changes.

therock-pr-bot · 2026-10-06T17:08:01Z

❌ PR Check — Action Required

Check	Status	Details
📝 PR Description	❌ Fail	Error: PR description must reference a JIRA ID, ISSUE ID, or a GitHub closing keyword. Expected: include a `JIRA ID` / `ISSUE ID` line (separator `:` or `-`, or omitted; value may be a JIRA key, a number with/without `#`, or a link), OR a closing keyword + issue reference. Accepted examples: • `JIRA ID : TESTAUTO-6039` • `JIRA ID - #330` • `JIRA ID #330` • `JIRA ID` (on separate line) `ROCM-25757` • `ISSUE ID : TESTUTO-3334` • `ISSUE ID #3334` • `ISSUE ID - TESTAUTO-3433` • `ISSUE ID` (on separate line) `AIRUNTIME-2352` • `ISSUE ID : https://github.com/<org_name>/<repo_name>/issues/1234` • `Closes #10` • `Fixes octo-org/octo-repo#100` • `Resolves: #123` • `#123` • `https://github.com/<org_name>/<repo_name>/issues/123` Current: no valid JIRA/ISSUE/closing-keyword reference found
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	⚠️ Warning	Error: Source/code files changed without an accompanying unit test. Expected: add at least one test file named like `test_<name>.py` / `test_<name>.cpp` (or `<name>_test.`). Current:* code file(s) changed: `projects/rccl/src/algorithms/dda/all_gather/all_gather_dda_fabric_ll128.h`, `projects/rccl/src/algorithms/dda/all_reduce/all_reduce_dda_ll128.h`, `projects/rccl/src/algorithms/dda/all_reduce/all_reduce_dda_ll128_twoshot.h`, `projects/rccl/src/algorithms/dda/device/CollCommon.h`, `projects/rccl/src/algorithms/dda/device/CollCommon_ll128.h` (+9 more); no test file found
🔎 pre-commit	⏳ Pending	⏳ Still running…
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

⚠️ 1 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

❌ PR Description

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-10-06T17:08:03Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

Copilot

Copilot review overview

🟡 Changes recommended

The LL128 tier is not selected under the default configuration, and IPC medium messages can be copied twice.

Review effort: Balanced
Findings: 1 · 1 · 1

Open (3)

LL128 eligibility range is shadowed by the LL tier · New
Host and kernel use inconsistent copy-size thresholds · New
Add MPI coverage for LL128 reduction kernel execution · New

What changed in this PR

Adds LL128 protocol support for DDA fabric reduce-scatter and optimizes IPC staging.

Changes:

Implements LL128 reduce-scatter packing, synchronization, and reduction.
Derives launch and scratch geometry dynamically.
Adds eligibility tests and medium-message IPC staging.

File	Description
`DdaFabricEligibilityTests.cpp`	Adds LL128 eligibility boundary tests.
`reduce_scatter_dda.h`	Adds in-kernel IPC staging.
`reduce_scatter_dda_fabric_ll128.h`	Implements the LL128 device kernel.
`dda_reduce_scatter_ipc.cu`	Selects IPC staging strategy.
`dda_reduce_scatter_fabric_ll128.cu`	Adds LL128 launch geometry and eligibility.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

amd-argus · 2026-10-06T17:21:36Z

@cppccppccppc @sunguobao PTAL

amd-argus

ARGUS

Bottom line: Worth settling before approval: ReduceScatterLL128_UnalignedBytes passes 4 bytes, which the old % 8 gate rejected too, so nothing in the suite pins the new 16-byte rule.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

amd-argus

ARGUS

Bottom line: Worth settling before approval: the ReduceScatter LL128 cap going 2 MiB to 128 MiB makes the rewritten ddaReduceScatterFabricLL128 the default path over a range it has never served, with no opt-in.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

amd-argus

ARGUS

Bottom line: Worth settling before approval: the gfx1250 default still routes reduce-scatter in (1 MiB, 64 MiB] onto the rewritten LL128 kernel, and the new MPI cases are registered only on gfx942 and gfx950 lanes where their gfx1250 gate always skips.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

amd-argus

ARGUS

Bottom line: Nothing new worth raising; still open from earlier: rccl_arch_thresholds.cc:44 and 2 more. Worth clearing before approval.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

nusislam · 2026-10-12T21:54:21Z

The code coverage CI failures are not related to this PR. There are failures for RMA, CE, DDA Alltoall and Allgather tests.

nusislam · 2026-10-12T22:28:12Z

@nawrinsu, @srirajpaul - Please review.

nusislam · 2026-10-16T13:38:20Z

I saw you put the copy kernel for dda based on the message size. What is the performance impact of doing that?

Moving the copy inside the kernel restores the DDA performance for the small messages (12us vs. 7us). The 12us latency was the effect of adding the done event tracking.

garzaran

@nusislam, I am approving the PR, but Praneeth has several comments about the tests. Please address those before merging.

nusislam · 2026-10-18T15:20:17Z

@copilot resolve the merge conflicts in this pull request

…-rs-ll128 # Conflicts: # projects/rccl/test/DdaReduceScatterMPITests.cpp Co-authored-by: nusislam <25847498+nusislam@users.noreply.github.com>

Copilot · 2026-10-18T15:27:51Z

@copilot resolve the merge conflicts in this pull request

Merged current develop and resolved the MPI test conflict while preserving both test suites in 094d6c604.

amd-argus

ARGUS

This PR changes 1,011 lines across 22 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Worth settling before approval: removing kDdaLL128RsMaxBytes leaves DdaReduceScatterMPITests.cpp:139 unable to compile, and that suite's odd-8B-word LL128 shapes now fail the new % 16 gate.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

amd-argus

ARGUS

This PR changes 1,524 lines across 23 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Worth settling before approval: the updated DdaReduceScatterMPITests.cpp landed at the monorepo root, so the built copy in projects/rccl/test still uses the removed kDdaLL128RsMaxBytes and rccl-UnitTestsMPI will not compile.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

amd-argus

ARGUS

This PR changes 1,090 lines across 23 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Nothing new worth raising; still open from earlier: rccl_arch_thresholds.cc:47 and 2 more. Worth clearing before approval.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

#12928 moved the fabric LL128 reduce-scatter to the warp-per-slice layout (512 threads, 16-byte shards and buffers, the LL128 cap, a slot sized from the scratch bank) and dropped DDA_LL128_RS_THREADS; the IPC launcher now copies shards up to 1 MiB in the kernel and takes the stop event before the host staging copy. Co-Authored-By: Claude <noreply@anthropic.com>

systems-assistant · 2026-10-21T13:32:06Z

TheRock Submodule Bump Activity

Newest first

2026-10-21 — Included in TheRock via ROCm/TheRock#9398.

nusislam added 4 commits October 5, 2026 11:56

add ll128 support for dda RS

751e916

move copy to temp buffer inside the kernel

f9d4e7c

change the dda RS threshold

7f55d41

update kernel-copy vs. memcpy threshold

72a57b7

nusislam requested review from a team and a balanced review from Copilot October 6, 2026 17:06

github-actions Bot added the project: rccl label Oct 6, 2026

Copilot started reviewing on behalf of nusislam October 6, 2026 17:08

View session

therock-pr-bot Bot added the Not ready to Review label Oct 6, 2026

Copilot AI reviewed Oct 6, 2026

View reviewed changes

Comment thread projects/rccl/src/algorithms/dda/reduce_scatter/dda_reduce_scatter_fabric_ll128.cu

Comment thread projects/rccl/src/algorithms/dda/reduce_scatter/dda_reduce_scatter_ipc.cu Outdated

Comment thread projects/rccl/test/DdaFabricEligibilityTests.cpp

systems-assistant Bot added the organization: ROCm label Oct 6, 2026

amd-argus Bot reviewed Oct 6, 2026

View reviewed changes

update thresholds and add tests

bc66253

nusislam changed the title ~~Add LL128 support for DDA RS~~ feat(rccl): Add LL128 support for DDA RS Oct 6, 2026

amd-argus Bot reviewed Oct 6, 2026

View reviewed changes

address review comments

4a042b3

nusislam added the ci: code-coverage label Oct 7, 2026

amd-argus Bot reviewed Oct 8, 2026

View reviewed changes

add new tests to the config files

a3d07a4

nusislam force-pushed the users/nusislam/dda-rs-ll128 branch from 97c0013 to a3d07a4 Compare October 8, 2026 14:08

amd-argus Bot reviewed Oct 8, 2026

View reviewed changes

nusislam requested review from nawrinsu and srirajpaul October 9, 2026 17:44