test(rccl): add host microtests for src/bootstrap.cc by aduforest · Pull Request #13601 · ROCm/rocm-systems

aduforest · 2026-10-15T22:02:22Z

JIRA ID: AICOMRCCL-2796

What

Host-only microtests for src/bootstrap.cc in a new rccl-UnitTestsMicroBootstrap binary: root rendezvous, socket and net rings, bootstrapSplit, P2P with parked connections, collectives, async encrypted sends, NCCL_OOB_NET_IFNAME and the abort/failure paths.

How

The test #includes the hipified bootstrap.cc. Ranks run as threads over an in-memory socket layer and net plugin in fakes/bootstrap_micro_fakes.cc, so the real protocol runs end to end; faults are injected per socket/plugin call.
Reuses fakes/os_fakes.cc (plus ncclOsSetFilesLimit and ncclOsStrSep), fakes/env_fakes.cc, fakes/hip_fakes.cc and the real misc/utils.cc. Own binary because bootstrap_stubs.cc fakes this unit for the others.
netGetDevice caches its device in a function-local static, so those cases are threadsafe death tests.

Pinned bugs

Pinned as KnownBug_ tests, not fixed (AICOMRCCL-3002..3004 for the first four):

bootstrapCreateRoot leaks the listen fd when ncclSocketGetAddr fails.
bootstrapInit and bootstrapSplit leak the proxy listen fd on any failure after the proxy listen.
bootstrapClose with parked connections and no abort leaves unexpectedConnections pointing at the freed list, so a second close frees it again.
A grow under NCCL_OOB_NET_ENABLE=1 + NCCL_BOOTSTRAP_BIDIR_NET=1 hangs until abort (the reverse ring never connects).
ncclSocketAccept(retry=false) reports a rejected peer as Accepting, so bootstrapAccept returns an unconnected socket and one stray connection makes the root exit. Open fix(rccl): stop TLS session tickets and stray peers from breaking bootstrap #13368 fixes this one.

Coverage

bootstrap.cc: 1476/1480 lines, 70/70 functions; the 4 missed lines are dead code.

Mutation Testing

46 targeted mutations, 45 killed; the survivor (rank < D to rank <= D in rootIdFromRank) is equivalent.

Test Plan

76 tests pass in about 2 s in the standalone build, and --gtest_shuffle --gtest_repeat=20 is clean.

Host-only microtests for src/bootstrap.cc in a new rccl-UnitTestsMicroBootstrap binary. Ranks run as threads over an in-memory socket layer and net plugin defined in the test, so the root rendezvous, the socket and net rings in both directions, split, grow, P2P send/recv and the async encrypted sends run for real. Known production bugs are pinned as observed, not fixed. Co-Authored-By: Claude <noreply@anthropic.com>

therock-pr-bot · 2026-10-15T22:03:10Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

aduforest · 2026-10-15T22:14:36Z

@argus review

amd-argus

ARGUS

Findings

5 more finding(s), not posted inline

projects/rccl/test/host/bootstrap-test.cc:1667 (Low) Split_EachSetupFailureIsReturned: fault lambda's loop-local std::map calls is also reached from the detached root thread
projects/rccl/test/host/bootstrap-test.cc:1743 (Low) Grow_NetBidir sets a 300 ms watchdog_ that also applies to the bidir=0 runs expected to succeed
projects/rccl/test/host/bootstrap-test.cc:1963 (Low) FailFirstOfTwoSends waits on gateCv_ and asyncSendCond with no deadline, so a regression hangs the binary
projects/rccl/test/host/bootstrap-test.cc:1210 (Low) Pinned production-bug tests carry no tracking ticket, so whoever fixes the bug sees an unexplained test failure
projects/rccl/test/host/bootstrap-test.cc:189 (Low) g_rejectedState default copies socket.cc's accept behaviour, so the #13368 pinned tests won't flip when socket.cc is fixed

This PR changes 2,094 lines across 8 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Worth settling before approval: Barrier_EveryRankLeavesAfterAllArrive only checks return codes, so a barrier that returns without waiting still passes.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

Hold one rank back in the barrier test so an early exit fails it, keep fault hooks out of reach of the root thread, bound the async-send waits, give the grow test's succeeding runs the full watchdog, and say in each pinned-bug comment what a fix flips instead of citing PR history. Co-Authored-By: Claude <noreply@anthropic.com>

aduforest · 2026-10-15T22:53:43Z

Addressed in 3e01295. On the non-inline findings:

:1667, root thread reaching the calls map: InitRanks now waits for its root thread to finish before returning. No fault hook installed after it can be reached from a root thread.
:1743, 300 ms watchdog on the bidir=0 runs: only the bidir runs, which are expected to hang, get 300 ms now. The bidir=0 runs keep the full wait limit.
:1963, unbounded waits in FailFirstOfTwoSends: both waits are now wait_for(g_waitLimit, ...) with an expectation, so a regression fails instead of hanging.
:1210, no tracking ticket on pinned bugs: this repo's microtests don't carry ticket refs in code, and all the pinned bugs are listed in the PR description. Each pinned test's comment now says it pins observed behaviour and which expectation a fix flips, so a fix won't trip on an unexplained failure.
:189, g_rejectedState copying socket.cc: the fix for this (fix(rccl): stop TLS session tickets and stray peers from breaking bootstrap #13368) changes bootstrapAccept in bootstrap.cc, which these tests compile. When it lands, the pinned tests flip as intended. If socket.cc itself changes instead, the fake's default has to follow it, and the comment on it now says it mirrors socket.cc.

@argus re-review

aduforest · 2026-10-15T22:53:49Z

@argus re-review

amd-argus

ARGUS

This PR changes 2,116 lines across 8 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Worth settling before approval: Root_RankGoneBeforeItsAnswer_MakesTheRootGiveUp and IntraNodeAllGather_EncryptedConnectsOnAHelperThread still pass with the behaviour they name removed.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

Assert the root leaves rank 1 unanswered once its send to rank 0 fails, and that the encrypted intra-node connects run on a thread other than the rank's. Co-Authored-By: Claude <noreply@anthropic.com>

aduforest · 2026-10-15T23:15:35Z

@argus re-review

amd-argus

ARGUS

This PR changes 2,137 lines across 8 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Worth settling before approval: connectThreads.size() at bootstrap-test.cc:1944 can come up short when helper thread ids are reused, so the encrypted allgather test can flake in pre-checkin.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

Compare each gathered proxy and P2P address with the listener the owning rank created, require distinct UDS ids, and mark rank threads with a thread_local so the helper-thread check does not depend on thread ids. Co-Authored-By: Claude <noreply@anthropic.com>

aduforest · 2026-10-15T23:38:57Z

@argus re-review

amd-argus

ARGUS

This PR changes 2,144 lines across 8 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM once BcastGrowHandle_CoordinatorReachesOnlyTheBoundaryRanks asserts nothing is pending on rank 2's peerSocket.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

Co-Authored-By: Claude <noreply@anthropic.com>

aduforest · 2026-10-16T00:00:54Z

@argus re-review

amd-argus

ARGUS

This PR changes 2,146 lines across 8 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

…comrccl-2796-bootstrap-host-microtests # Conflicts: # projects/rccl/test/host/MICROTEST_README.md # projects/rccl/test/host/run_host_tests.sh # projects/rccl/tools/scripts/test_runner/configs/ci-precheckin.json # projects/rccl/tools/scripts/test_runner/lib/test_executor.py

aduforest · 2026-10-16T17:11:34Z

@argus review

amd-argus

ARGUS

This PR changes 2,146 lines across 8 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

- NetMultiOp_RejectsEachMalformedOpBeforePostingAny counts isend/irecv on a real loopback comm instead of g_netConns, which no op could change. - Split_EachSetupFailureIsReturned closes each iteration's parent and child states (Split overwrote parents_, leaking them) and checks no listener is left. - TearDown resets the seams; RAS_ENABLE goes through g_params; the stagger test uses a real delay and checks it. - Pinned production bugs are named <Fn>_KnownBug_<What> with one-line comments. Co-Authored-By: Claude <noreply@anthropic.com>

aduforest · 2026-10-16T20:21:47Z

@argus review

amd-argus

ARGUS

This PR changes 2,150 lines across 8 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

…comrccl-2796-bootstrap-host-microtests # Conflicts: # projects/rccl/test/host/CMakeLists.txt # projects/rccl/test/host/MICROTEST_README.md # projects/rccl/tools/scripts/test_runner/configs/ci-precheckin.json # projects/rccl/tools/scripts/test_runner/lib/test_executor.py

aduforest · 2026-10-19T17:59:22Z

@argus review

amd-argus

ARGUS

This PR changes 2,178 lines across 8 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

- The in-memory socket layer, net plugin, logging, params and the proxy/RAS seams move to fakes/bootstrap_micro_fakes.{h,cc}, listed in the source list and the README map. - getHostHash, getPidHash, hashCombine, parseStringList and matchIfList are the real misc/utils.cc now (with hip_fakes.cc, and ncclOsStrSep in os_fakes.cc). - The OOB-device death tests restore death_test_style. - KnownBug_ pins name their Jira ticket; the yaml no longer claims sub-second. - test_executor.py lists only this PR's binary. Co-Authored-By: Claude <noreply@anthropic.com>

…comrccl-2796-bootstrap-host-microtests

aduforest · 2026-10-19T20:35:54Z

@argus review

amd-argus

ARGUS

This PR changes 2,243 lines across 10 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

github-actions Bot added the project: rccl label Oct 15, 2026

aduforest changed the title ~~test(rccl): add host microtests for the bootstrap~~ test(rccl): add host microtests for src/bootstrap.cc Oct 15, 2026

systems-assistant Bot added the organization: ROCm label Oct 15, 2026

amd-argus Bot reviewed Oct 15, 2026

View reviewed changes

Comment thread projects/rccl/test/host/bootstrap-test.cc Outdated

Comment thread projects/rccl/test/host/bootstrap-test.cc Outdated

amd-argus Bot reviewed Oct 15, 2026

View reviewed changes

Comment thread projects/rccl/test/host/bootstrap-test.cc

Comment thread projects/rccl/test/host/bootstrap-test.cc Outdated

test(rccl): check the root and the TLS helper thread directly

45917fa

Assert the root leaves rank 1 unanswered once its send to rank 0 fails, and that the encrypted intra-node connects run on a thread other than the rank's. Co-Authored-By: Claude <noreply@anthropic.com>

amd-argus Bot reviewed Oct 15, 2026

View reviewed changes

Comment thread projects/rccl/test/host/bootstrap-test.cc Outdated

Comment thread projects/rccl/test/host/bootstrap-test.cc

amd-argus Bot reviewed Oct 15, 2026

View reviewed changes

Comment thread projects/rccl/test/host/bootstrap-test.cc

test(rccl): check bcastGrowHandle leaves the inner ranks alone

517565d

Co-Authored-By: Claude <noreply@anthropic.com>

amd-argus Bot reviewed Oct 16, 2026

View reviewed changes

aduforest marked this pull request as ready for review October 16, 2026 00:19

aduforest requested review from a team October 16, 2026 00:19

amd-argus Bot reviewed Oct 16, 2026

View reviewed changes

amd-argus Bot reviewed Oct 19, 2026

View reviewed changes

aduforest and others added 2 commits October 19, 2026 13:35

Merge remote-tracking branch 'origin/develop' into users/aduforest/ai…

ad00342

…comrccl-2796-bootstrap-host-microtests

amd-argus Bot reviewed Oct 19, 2026

View reviewed changes

Conversation

aduforest commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Pinned bugs

Coverage

Mutation Testing

Test Plan

Uh oh!

therock-pr-bot Bot commented Oct 15, 2026

✅ All Policy Checks Passed

Uh oh!

aduforest commented Oct 15, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Findings

Uh oh!

Uh oh!

Uh oh!

aduforest commented Oct 15, 2026

Uh oh!

aduforest commented Oct 15, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aduforest commented Oct 15, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aduforest commented Oct 15, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aduforest commented Oct 16, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

aduforest commented Oct 16, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

aduforest commented Oct 16, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

aduforest commented Oct 19, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

aduforest commented Oct 19, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

aduforest commented Oct 15, 2026 •

edited

Loading