Skip to content

ci(rccl): run Anvil plugin unit tests in GIN sbatch - #12441

Closed
dlamd1dai wants to merge 1 commit into
ROCm:developfrom
dlamd1dai:users/dondai/aicomrccl-2339-anvil-plugin-ci
Closed

dlamd1dai wants to merge 1 commit into
ROCm:developfrom
dlamd1dai:users/dondai/aicomrccl-2339-anvil-plugin-ci

Conversation

@dlamd1dai

Copy link
Copy Markdown
Contributor

Motivation

Follow-up to #11569 / AICOMRCCL-2339. gin.sbatch already builds rccl-UnitTestsGinAnvilPlugin under GIN_ANVIL_UNIT_TESTS=ON, but only the SDMA factory gtest is executed. Logical-context bind coverage therefore has no automated GIN-CI run.

Technical Details

  • Export RCCL_ANVIL_PLUGIN_TEST from the build stage
  • Run the plugin binary after the SDMA factory gtest (non-gating, matching factory)

JIRA ID: AICOMRCCL-2339

Test Plan

  • GIN CI single-node job builds and executes rccl-UnitTestsGinAnvilPlugin

Test Result

  • Pending GIN CI on this PR

Submission Checklist

Made with Cursor

gin.sbatch already builds rccl-UnitTestsGinAnvilPlugin under
GIN_ANVIL_UNIT_TESTS=ON; execute it next to the SDMA factory gtest so
logical-context bind coverage is exercised in CI (AICOMRCCL-2339).

Co-authored-by: Cursor <cursoragent@cursor.com>
@dlamd1dai
dlamd1dai requested review from a team September 29, 2026 03:34
@therock-pr-bot

Copy link
Copy Markdown

✅ All Policy Checks Passed

Check Status Details
📝 PR Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass PR does not contain code files — Unit Test auto-passed
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@amd-argus amd-argus Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ARGUS

Findings

1 finding(s) that could not be posted as inline comments
  • projects/rccl/tools/ci/lib/gin.sbatch:23 (Low) Header step map still ends step 6 at the SDMA factory gtest and omits the new 6b run The header step map at projects/rccl/tools/ci/lib/gin.sbatch:23 still ends step 6 at rccl-UnitTestsGinAnvilSdmaFactory, so it no longer lists the Anvil plugin gtest that now runs as 6b. A line next to the step 6 entry keeps the header an accurate map of the script.

Bottom line: Worth settling before approval: this binary already runs in step 7 via gin-tests.json, so the gap the PR sets out to close may not exist.

Comment @argus review to re-run, or @argus re-review after pushing.


# --- unit test: Anvil host plugin gtest (already built in step 4). ---
# Covers bind-signals / logical-context stripe coverage that install.sh -t skips.
echo "==> [6b/7] run Anvil plugin unit tests"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium

Step 7 already runs this binary, so the stated gap does not hold: projects/rccl/tools/ci/lib/gin-tests.json:58 registers rccl-UnitTestsGinAnvilPlugin as rccl-gin-conn-check-plugin-ut, and projects/rccl/tools/ci/run-gin-ci.sh:202 resolves that to the same ${rccl_build}/test path this line uses. That run is also timeout wrapped and feeds the tests-failed sentinel, while this one only echoes to stderr. Could this step be dropped?

"${anvil_plugin_test}" --gtest_output="xml:${logs_dir}/anvil-plugin-test.xml" \
|| echo "WARNING: Anvil plugin unit tests failed (non-gating)" >&2
else
echo " SKIP: ${anvil_plugin_test} not found (GIN_ANVIL_UNIT_TESTS may be off)"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low

This SKIP branch cannot be reached: line 227 already exits 1 when the plugin binary is not executable, and line 303 skips the whole test phase whenever the build failed. Its hint is wrong too, since projects/rccl/test/CMakeLists.txt:466 gates this target on ENABLE_ROCSHMEM_GIN and ROCM_VERSION >= 60400, not on GIN_ANVIL_UNIT_TESTS.

[[ -d "${_sp}" ]] && _sysdeps_lib="${_sp}"
done
export LD_LIBRARY_PATH="${RCCL_INSTALL_PREFIX}/lib:${ROCSHMEM_INSTALL_DIR}/lib:${ROCM_PATH}/lib${_sysdeps_lib:+:${_sysdeps_lib}}:${LD_LIBRARY_PATH:-}"
"${anvil_plugin_test}" --gtest_output="xml:${logs_dir}/anvil-plugin-test.xml" \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low

This gtest runs with no wall clock cap, unlike every fixtures entry in projects/rccl/tools/ci/run-gin-ci.sh:255, which wraps the same binary in timeout --kill-after. A stall here would burn the job's --time=04:00:00 budget before step 7 ever starts. Could it be wrapped the same way?

for _sp in "${ROCM_PATH}/lib/rocm_sysdeps/lib" "${ROCM_PATH}/core/lib/rocm_sysdeps/lib"; do
[[ -d "${_sp}" ]] && _sysdeps_lib="${_sp}"
done
export LD_LIBRARY_PATH="${RCCL_INSTALL_PREFIX}/lib:${ROCSHMEM_INSTALL_DIR}/lib:${ROCM_PATH}/lib${_sysdeps_lib:+:${_sysdeps_lib}}:${LD_LIBRARY_PATH:-}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low

Lines 341-345 are byte identical to 325-329 in the SDMA factory block, and the rest of this block mirrors 320-334 with only the names swapped. A helper taking binary, xml name and label would collapse both sites and stop them drifting; it needs to sit at top level, since the build subshell that holds the other helpers ends at line 299.

printf 'export RCCL_TESTS_BIN_DIR=%q\n' "${rccl_tests_build}"
printf 'export RCCL_TESTS_DIR=%q\n' "${rccl_tests_src}"
printf 'export RCCL_FIXTURES_BIN_DIR=%q\n' "$(dirname "${rccl_fixtures_bin}")"
printf 'export RCCL_ANVIL_PLUGIN_TEST=%q\n' "${rccl_conn_plugin_bin}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low

RCCL_ANVIL_PLUGIN_TEST duplicates the fallback that consumes it: RCCL_FIXTURES_BIN_DIR is the dirname of line 121, the same ${rccl_build}/test directory line 122 builds this path from, so both branches of line 339 resolve to the same file. It is not an override hook either, since line 316 re-sources rccl.env last and overwrites any inherited value. Could this use the plain form line 323 uses?

@dlamd1dai

Copy link
Copy Markdown
Contributor Author

Superseded by #12745 and closing in favour of it. Same commits, same content; only the head branch moved.

Why. This PR was opened from a fork, and the required RCCL PR gate check can never pass that way. Fork pull requests get no secrets, so the RCCL coco PR workflow skips its Setup job, the coco smoke never runs, and the gate fails closed within seconds. Clearing it here would need a maintainer to dispatch the workflow by hand and then override the check, on every push. #12745 is branched in ROCm/rocm-systems, where Setup succeeds and coco dispatches to both the Ruby and OCI clusters on its own.

Please keep reading this thread. 5 of the 5 review threads here are still unresolved, and inline conversations do not carry across to a new pull request. Argus will re-review #12745 and should re-surface anything still unfixed, but the replies and reasoning written here only exist on this PR, so check them before merging the replacement.

No further work is needed on this branch. Closing rather than leaving it open so it does not keep consuming CI or show up as a duplicate for reviewers; it stays readable and can be reopened.

@dlamd1dai dlamd1dai closed this Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant