Repository navigation
Conversation
The bind walk read g_pendingByComm without the lock that add, remove, and clear hold. Copy the list under the mutex so registration and clear can take it. Co-authored-by: Cursor <cursoragent@cursor.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
There was a problem hiding this comment.
ARGUS
Findings
1 finding(s) that could not be posted as inline comments
projects/rccl/src/gin/gin_plugin_anvil_sdma.cc:90(Low) ginAnvilPendingRemove leaves a permanent empty map node per destroyed contextginAnvilPendingRemoveat gin_plugin_anvil_sdma.cc:90 readsg_pendingByComm[comm], which re-inserts a{comm, nullptr}node whenever the bind path has already erased the entry at line 836, and it never erases the node when it unlinks the last context. EveryginAnvilDestroyContext(line 956) therefore leaves one entry behind, keyed by anncclComm*that is freed shortly after, so the map grows once per comm for the life of the process. This predates the PR and the samefind()shape you used at line 802 would close it, so it seems better as a separate change than folded in here.
Bottom line: Looks approval ready.
Comment @argus review to re-run, or @argus re-review after pushing.
|
Superseded by #11569 (AICOMRCCL-2339). #11569 was rebased onto current Closing in favour of #11569. |
> **Replaces #11569.** Identical commits and content; head is `cf8cf9720f16391c76a19534b2af34eaea0c67cd` on both. > > #11569 was opened from a fork, where the required `RCCL PR gate` check can never pass. Fork pull requests receive no secrets, so the `RCCL coco PR` workflow skips its `Setup` job, the coco smoke never runs, and the gate fails closed within seconds. Clearing it there needs a maintainer to dispatch the workflow by hand and override the check, on every push. This branch lives in `ROCm/rocm-systems`, so the gate dispatches coco to the Ruby and OCI clusters on its own. > > Review history stays on #11569: 27 threads, all resolved. No approval was lost; it had no approval or changes-requested decision. ## Motivation Anvil SDMA currently allocates one GPU context per connection and ignores `nContexts`. When `ncclGinAllContexts` barriers run on a shared connection, every logical context writes and waits on the same signal cells, so a later barrier can complete on a stale count from an earlier one. ## Technical Details Give each logical context its own `ncclGinAnvilSdmaGPUContext` and LSA signal stripe while keeping SDMA queues, dirty bits, and counters shared at the connection. Device APIs now index `ctx.handle` by `ctx.contextId`. Resource-window bind walks `nContexts` slots instead of one slot per plugin context. Signal peer addresses are indexed in GIN-team space (`ctx->rank`), matching `resolveRemotePeerVa`; `regMrSym` keeps `devr->lsaSelf` because that table is LSA-indexed. The signal span is registered in the IPC table once per bind rather than once per context, and the comm-wide collectives (the local-base allgather and the LSA conn-check) run at span scope so a comm with N logical contexts still costs one bootstrap round trip, not N. Two fixes to the surrounding code are folded in, previously proposed standalone as #12450 and #12451 and closed in favour of this PR once the restructuring above made them inapplicable as separate patches: - The span base probe is sized with `comm->nRanks` and publishes the local VA at `comm->rank`. `bootstrapAllGather` writes one slot per world rank, so a GIN-team-sized buffer was overrun and the VA landed in the wrong slot under `NCCL_GIN_CONNECTION_RAIL`. - The bind path snapshots `g_pendingByComm` into a vector under `pluginMutex` and walks the copy. Its three reads (span geometry, slot binding, and `fail:` rollback) were unlocked while `ginAnvilPendingAdd`/`Remove`/`Clear` all hold that non-recursive mutex; `find()` also avoids default-inserting an empty list for a comm with nothing pending. JIRA ID: AICOMRCCL-2339 AICOMRCCL-3072 ## Test Plan - Host plugin unit tests, including `BindSignals_LogicalContextsUseDistinctSignalStripes` and the `ConnCheck_*` suite - Device template tests, including `SignalApis_SelectLogicalContext` and `PutValue_SelectLogicalContext` - MPI regression `Barrier_AllContextsConsecutiveSignalsDoNotAlias_SingleNode` (Anvil, 1 node / 2 ranks, `NCCL_GIN_TYPE=7`, delayed second barrier) ## Test Result Anvil SDMA is `NCCL_GIN_TYPE=7` (`NCCL_NET_DEVICE_GIN_ANVIL_SDMA`). Earlier runs in this section were recorded when the enum was 6. Pre-rebase (commit `970a014315`): - `rccl-UnitTestsGinAnvilPlugin`: 20/20 passed on MI355 - `GinAnvilSdmaTemplateTest.*`: 13/13 passed on MI355 (before `PutValue_SelectLogicalContext` was added) - `GinMPIDeviceTests.Barrier_AllContextsConsecutiveSignalsDoNotAlias_SingleNode` on MI355: **PASSED** (7779 ms) on `smci355-ccs-aus-m03-17`, 1 node / 2 ranks, `NCCL_GIN_ENABLE=1`, `NCCL_CUMEM_ENABLE=1`. Ran `rccl-UnitTestsMPI` inside `rccl-gin-pr11556-rs:latest` with `--ipc=host`. Host `mpirun` on the same node segfaulted in HIP IPC during communicator bring-up (`ncclP2pImportShareableBuffer` / `ncclDevrCommCreateInternal`) while other GPU docker jobs held `/dev/kfd` and `/dev/dri`; that path never reached the kernel. - Same MPI filter on MI350: **PASSED** (8834 ms) on `cv350-rck-g03-e23-08`, 1 node / 2 ranks, same GIN env. Host `mpirun -np 2`; GPUs were idle. Used the gfx950 `rccl-UnitTestsMPI`/`librccl.so.1` from the MI355 PR build (native CentOS 9 link of `librccl.so` failed on `librocshmem_gin_host.a` bitcode encoding; Docker was unavailable on that node). Current head `cf8cf9720f`, rebased onto `develop` at `482e541856`, retested from a clean worktree on MI355X (`smci355-ccs-aus-m03-17`), 1 node / 2 ranks, `NCCL_GIN_TYPE=7`, `NCCL_GIN_ENABLE=1`, `NCCL_CUMEM_ENABLE=1`, `NCCL_GIN_NCONTEXTS=4`: - `rccl-UnitTestsGinAnvilPlugin`: **37/37 passed** (304 ms), including the full `ConnCheck_*` suite and the two span-scope cases added for the ARGUS finding. Both were confirmed to fail against `8eafd3d752` (3 allgathers at `nContexts=3`) and pass here (1). - Whole `GinMPIDeviceTests.*` suite on the Anvil backend: **44 passed, 20 skipped, 0 failed** (40.1 s). Covers `Barrier_AllContextsConsecutiveSignalsDoNotAlias_SingleNode` (680 ms), all seven `BarrierFence_*_SingleNode` cases including `AllContextsPut`/`AllContextsGet`, `MultiContext_AllFourRoute`/`NonPowerOf2`/`Exclusive`, and `Put_BasicAndOffsets_MultiContext`/`PutValue_Inline_MultiContext`/`WaitCounterAndSignal_MultiContext`. - The 20 skips are topology or opt-in gated, not regressions: cross-node and 4-rank cases, `RailConnection_Create`, the `VASignal_*` cases (VA signals are unsupported on SDMA by design), and the pool tests that require `NCCL_GIN_SIGNAL_POOL_SIZE>=0x40000000`. An earlier attempt at this run skipped all 64 cases because `NCCL_CUMEM_ENABLE` was unset, which trips the `Symmetric memory required` gate; the numbers above are from the corrected invocation. Earlier post-rebase run on `f32754f6c1`, same node and env, before #12450/#12451 were folded in: plugin **37/37**, `GinMPIDeviceTests.*` **39 passed, 23 skipped, 0 failed** (34.5 s). Environment note: the host ROCm on that node is 7.0.2.1 while its KFD driver is newer, and `hipIpcOpenMemHandle` segfaults during plain P2P transport setup (`ncclP2pImportShareableBuffer`) before any GIN code runs. The runs above use the node's ROCm 7.13 runtime in a container with `--ipc=host`, where the same binaries pass. Not related to this change; the same workaround was needed for the pre-rebase runs. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Motivation
Follow-up to #11569 / AICOMRCCL-2339.
ncclGinAnvilBindResourceWindowSignalswalkedg_pendingByComm[comm]with no lock.ginAnvilPendingAdd,ginAnvilPendingRemove, andginAnvilPendingClearall takepluginMutex.operator[]also default-inserts an empty list when the comm has nothing pending.std::mutexis not recursive, so the walk cannot hold the lock acrossginAnvilRegisterLsaSignalsorginAnvilPendingClear, both of which lock it again.Technical Details
findthe comm underpluginMutexand copy the pendingginAnvilGinCtxpointers#11569 rewrites the bind path and still walks
g_pendingByCommunlocked (the span scan, the slot loop, and the failure path). If that PR merges first, rebase this change onto it.JIRA ID: AICOMRCCL-2339
Test Plan
NCCL_GIN_TYPE=7, 1 node / 2 ranks) still binds the pending contexts and clears the listTest Result
Submission Checklist
Made with Cursor