Skip to content

ci(rccl): run Anvil plugin unit tests in GIN sbatch - #12745

Closed
dlamd1dai wants to merge 2 commits into
developfrom
users/dondai/aicomrccl-2339-anvil-plugin-ci
Closed

dlamd1dai wants to merge 2 commits into
developfrom
users/dondai/aicomrccl-2339-anvil-plugin-ci

Conversation

@dlamd1dai

Copy link
Copy Markdown
Contributor

Replaces #12441. Identical commits and content; only the head branch moved.

#12441 was opened from a fork, where the required RCCL PR gate check can never pass. Fork pull requests receive no secrets, so the RCCL coco PR workflow skips its Setup job, the coco smoke never runs, and the gate fails closed in a few seconds. Clearing it there needs a maintainer to dispatch the workflow by hand and then override the check. This branch lives in ROCm/rocm-systems, so the gate dispatches coco on its own.

5 unresolved review thread(s) remain on #12441 (of 5 total). Read them there before merging this one; the inline conversations and any replies to them do not carry across to a new pull request.

Motivation

Follow-up to #11569 / AICOMRCCL-2339. gin.sbatch already builds rccl-UnitTestsGinAnvilPlugin under GIN_ANVIL_UNIT_TESTS=ON, but only the SDMA factory gtest is executed. Logical-context bind coverage therefore has no automated GIN-CI run.

Technical Details

  • Export RCCL_ANVIL_PLUGIN_TEST from the build stage
  • Run the plugin binary after the SDMA factory gtest (non-gating, matching factory)

JIRA ID: AICOMRCCL-2339

Test Plan

  • GIN CI single-node job builds and executes rccl-UnitTestsGinAnvilPlugin

Test Result

  • Pending GIN CI on this PR

Submission Checklist

Made with Cursor

gin.sbatch already builds rccl-UnitTestsGinAnvilPlugin under
GIN_ANVIL_UNIT_TESTS=ON; execute it next to the SDMA factory gtest so
logical-context bind coverage is exercised in CI (AICOMRCCL-2339).

Co-authored-by: Cursor <cursoragent@cursor.com>
@dlamd1dai
dlamd1dai requested review from a team October 3, 2026 20:15
@therock-pr-bot

therock-pr-bot Bot commented Oct 3, 2026

Copy link
Copy Markdown

✅ All Policy Checks Passed

Check Status Details
📝 PR Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass PR does not contain code files — Unit Test auto-passed
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@amd-argus amd-argus Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ARGUS

Bottom line: Worth settling before approval: gin-tests.json:57-63 already runs rccl-UnitTestsGinAnvilPlugin via run-gin-ci.sh in step 7, so this block re-runs the same binary with no timeout and no failure accounting.

Comment @argus review to re-run, or @argus re-review after pushing.

[[ -d "${_sp}" ]] && _sysdeps_lib="${_sp}"
done
export LD_LIBRARY_PATH="${RCCL_INSTALL_PREFIX}/lib:${ROCSHMEM_INSTALL_DIR}/lib:${ROCM_PATH}/lib${_sysdeps_lib:+:${_sysdeps_lib}}:${LD_LIBRARY_PATH:-}"
"${anvil_plugin_test}" --gtest_output="xml:${logs_dir}/anvil-plugin-test.xml" \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium

This invocation has no wall-clock cap, unlike every other test run in this job: run-gin-ci.sh:255 wraps each fixtures run in timeout with BENCH_TIMEOUT (600s by default). Because this runs before step 7, a wedged run consumes the 4h --time budget at line 7 and slurm kills the job, so lines 361-367 never write tests-failed and the hang reads as an infra failure instead of a test timeout. Could we wrap it the same way?

Comment thread projects/rccl/tools/ci/lib/gin.sbatch Outdated
printf 'export RCCL_TESTS_BIN_DIR=%q\n' "${rccl_tests_build}"
printf 'export RCCL_TESTS_DIR=%q\n' "${rccl_tests_src}"
printf 'export RCCL_FIXTURES_BIN_DIR=%q\n' "$(dirname "${rccl_fixtures_bin}")"
printf 'export RCCL_ANVIL_PLUGIN_TEST=%q\n' "${rccl_conn_plugin_bin}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low

RCCL_ANVIL_PLUGIN_TEST has no consumer and no effect: line 122 makes it ${rccl_build}/test/rccl-UnitTestsGinAnvilPlugin, which is exactly what the :- fallback at line 339 expands to through RCCL_FIXTURES_BIN_DIR. It cannot serve as an operator override either, since line 316 re-sources rccl.env over any inherited value before line 339 reads it. Could we drop it and locate the binary from RCCL_FIXTURES_BIN_DIR the way line 323 does?

Comment thread projects/rccl/tools/ci/lib/gin.sbatch Outdated
"${anvil_plugin_test}" --gtest_output="xml:${logs_dir}/anvil-plugin-test.xml" \
|| echo "WARNING: Anvil plugin unit tests failed (non-gating)" >&2
else
echo " SKIP: ${anvil_plugin_test} not found (GIN_ANVIL_UNIT_TESTS may be off)"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low

This SKIP names a cause the script forecloses: line 227 hard-fails the build when rccl_conn_plugin_bin is not executable and line 303 exits before reaching here, so the branch is unreachable. GIN_ANVIL_UNIT_TESTS is not what gates the target either; test/CMakeLists.txt:466 keys it on ENABLE_ROCSHMEM_GIN and ROCM_VERSION >= 60400.

Comment thread projects/rccl/tools/ci/lib/gin.sbatch Outdated
fi

# --- unit test: Anvil host plugin gtest (already built in step 4). ---
# Covers bind-signals / logical-context stripe coverage that install.sh -t skips.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit

Suggested change
# Covers bind-signals / logical-context stripe coverage that install.sh -t skips.
# Covers the bind-signals and conn-check cases in the Anvil host plugin gtest.

Give the early GIN sbatch run the same timeout and tests-failed accounting as the fixtures suite, and drop the unused path export and unreachable skip.

Co-authored-by: Cursor <cursoragent@cursor.com>
@dlamd1dai

Copy link
Copy Markdown
Contributor Author

Closing this. rccl-UnitTestsGinAnvilPlugin is already executed by step 7 as rccl-gin-conn-check-plugin-ut in projects/rccl/tools/ci/lib/gin-tests.json (no gtest filter, so the bind-signals and conn-check cases are included). That fixtures run is already wrapped in timeout and feeds the tests-failed sentinel.

Step 6b only runs the same binary again. The SDMA factory early run is different: rccl-UnitTestsGinAnvilSdmaFactory is not in the matrix. This plugin binary is.

@dlamd1dai dlamd1dai closed this Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant