ci(rccl): add Llama-3.1-70B PP=4 to the weekly madengine runs by mkuznet1 · Pull Request #13728 · ROCm/rocm-systems

mkuznet1 · 2026-10-19T10:36:59Z

Motivation

The madengine gate trains Llama-3.1-70B with FSDP2, where RCCL carries
all-gather / reduce-scatter. This adds the same model and image with the
parallel layout switched to TP1 x PP4 x DP4, so pipeline send/recv between
nodes is what RCCL carries. It runs on the weekly schedule only.

Stacked on #13182, which is stacked on #13180 and #12899.

Changes

New workload llama-3.1-70b-pp4-training. It reuses the 70B model, image,
precisions (BF16, FP8) and 4 h time_limit, and adds one env_vars entry:
PRIMUS_EXTRA_ARGS="--pipeline_model_parallel_size 4 --use_torch_fsdp2 false --use_distributed_optimizer true".
Megatron rejects FSDP2 together with PP > 1, so all three overrides go
together. required_nodes: 2, because PP4 on 16 GPUs is DP4 and another
scale would be another layout.
MAD_REF moves to 36339bd8, the Add PRIMUS_EXTRA_ARGS and MOE_A2A_BACKEND for Llama-3.1-70B PP=4 and DeepSeek-R1 a2a none MAD#284 merge on mad-rccl. That is
the only MAD change since 7a9a25e, and it is where the Megatron script
starts reading PRIMUS_EXTRA_ARGS.
check_mad_knobs() runs right after madengine install. It fails the run if
the pinned Megatron script never mentions a PRIMUS_* knob the workload
sets. Without it, an older pin would silently ignore the knob, and the PP=4
key would record FSDP numbers.
No separate madengine jobs. Every scheduled workload states its own nodes
and job_timeout_minutes. That holds for the nightly job and for each entry of
the weekly matrix:
- The weekly matrix gains the PP=4 entry (2 nodes, 390 minutes).
- The separate 405B job therock-test-madengine-405b from feat(rccl): add JAX/MaxText Llama-3.1-405B on 4 nodes to the madengine gate #13182 becomes a
  matrix entry (4 nodes, 450 minutes, rationale kept as a comment).
  max-parallel: 1 now serialises every weekly workload, 405B included.
  Before, 405B could queue for 4 exclusive nodes alongside a 2-node leg.
- The nightly job states job_timeout_minutes: '390' instead of inheriting
  the default.
Dispatch choice added in therock-rccl-test-madengine.yml.
tests/test_workload_configs.py checks that:
- the dispatch choices equal WORKLOAD_CONFIGS;
- every scheduled leg names a config and uses its required_nodes;
- every scheduled leg leaves the script more than its SLURM time_limit;
- check_mad_knobs behaves correctly against a script with and without the
  knob.

Nightly vs weekly after this stack

Schedule	Workloads
Nightly (Sun-Fri cron)	`llama-3.1-70b-training` (FSDP2, 2N)
Weekly (Sat cron, `test_scope=all`), serial	`gpt-oss-120b-training` (2N), `llama-3.1-70b-pp4-training` (2N), `llama-3.1-405b-maxtext` (4N)

Validation

python3 -m unittest discover -s tests from projects/rccl/ci/scripts:
19 tests pass.
check_mad_knobs against the real MAD scripts: the PP=4 workload passes on
36339bd8 and fails on 7a9a25e; gpt-oss-120b-training passes on both.
Same layout, image and MAD change run by hand on ruby (SLURM job 74493,
2 x 8 MI355X, one phase in 62 min, all SUCCESS):

Precision tok/s/GPU TFLOPS/GPU

BF16 1518.9 682.5

FP8 1658.7 745.4
CI dispatch of the new key: TODO. The dispatch from this branch,
run 39238797725,
failed with [Errno 122] Disk quota exceeded in "Create Python environment",
before any allocation: the runner account's home was full. feat(rccl): score the madengine gate against a same-node baseline #12899 now keeps
conda and pip out of that home.

amd-argus

ARGUS

Findings

5 more finding(s), not posted inline

projects/rccl/ci/scripts/test_madengine.py:318 (Low) check_mad_knobs passes if a knob name appears anywhere in the script text, including a comment
projects/rccl/ci/scripts/test_madengine.py:590 (Low) squash_log_spam drops the whole physical line on an unanchored match, losing any output glued to the warning
projects/rccl/ci/scripts/test_madengine.py:572 (Low) squash_log_spam, Step 5a and test_log_squash.py are not mentioned in the PR description
projects/rccl/ci/scripts/tests/test_workload_configs.py:3 (Low) No CI workflow runs projects/rccl/ci/scripts/tests, so the new workflow/config consistency checks guard nothing on PRs
projects/rccl/ci/scripts/tests/test_workload_configs.py:79 (Low) test_job_budget_covers_time_limit accepts 1 minute of queue slack, so 405b at job_timeout_minutes '390' still passes

Bottom line: Looks approval ready.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

amd-argus · 2026-10-19T11:40:18Z

+# at WARN, so NCCL_DEBUG=WARN keeps it. GPT-OSS-120B's MoE all-to-all made it
+# 777600 lines per node per phase -- 97% of a 180 MB node log, which madengine
+# keeps three copies of -- and run 38238634668 alone left 2.1 GB on /apps, under
+# the runner account's quota.


Low

A few new comments retell history (run ids, MB counts, throughput) that the workflow comments or MAD PR already record. Other sites: test_madengine.py:135-140, therock-rccl-ci-linux.yml:266-269. For example:

Suggested change

# the runner account's quota.

# RCCL prints this at WARN on every call of a deprecated API name

# (src/misc/api_trace.cc), so MoE all-to-all fills the node logs with it.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

therock-pr-bot · 2026-10-19T15:30:37Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	PR does not contain code files — Unit Test auto-passed
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

amd-argus

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

#13728 is stacked on this branch and already folds this job into the weekly matrix, with the same nodes '4' / job_timeout_minutes '450' and a third workload besides. Doing it here too just hands that PR a conflict to resolve against itself. Restores therock-test-madengine-405b as its own job and takes the per-entry job_timeout_minutes back out of the matrix, so the fold lands once, where it was planned.

amd-argus

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

New workload llama-3.1-70b-pp4-training: the 70B Primus/Megatron model on the same image, with FSDP2 swapped for TP1 x PP4 x DP4 via PRIMUS_EXTRA_ARGS, so pipeline point-to-point traffic is what RCCL carries. Fixed at 2 nodes, where the layout was validated (ROCm/MAD#284). - MAD_REF -> 36339bd8 (mad-rccl with MAD#284, which reads PRIMUS_EXTRA_ARGS). check_mad_knobs() fails the run before any allocation when the pinned Megatron script never mentions a PRIMUS_* knob the workload sets; otherwise the knob is ignored and the PP=4 key would score the FSDP layout. - The weekly matrix gains the PP=4 entry and absorbs the separate 405B job: every entry carries its own nodes and job_timeout_minutes, so one matrix with max-parallel: 1 serialises every weekly workload, 405B included. The nightly job states its 390 minutes too instead of inheriting the default. - tests/test_workload_configs.py: dispatch choices equal WORKLOAD_CONFIGS, every scheduled leg names a config, uses its required_nodes, and leaves the script more than its SLURM time_limit. Co-authored-by: Cursor <cursoragent@cursor.com>

amd-argus

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

…ama-70b-pp4

…ama-70b-pp4 Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # .github/workflows/therock-rccl-ci-linux.yml

amd-argus

ARGUS

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

mkuznet1 mentioned this pull request Oct 19, 2026

ci(rccl): add DeepSeek-R1 SGLang 2P+2D to the weekly madengine runs #13729

Open

github-actions Bot added github actions Pull requests that update GitHub Actions code project: rccl labels Oct 19, 2026

systems-assistant Bot added the organization: ROCm label Oct 19, 2026

mkuznet1 marked this pull request as ready for review October 19, 2026 11:25

mkuznet1 requested review from a team as code owners October 19, 2026 11:25

mkuznet1 mentioned this pull request Oct 19, 2026

feat(rccl): add JAX/MaxText Llama-3.1-405B on 4 nodes to the madengine gate #13182

Open

1 task done

mkuznet1 added this pull request to stack #13181 October 19, 2026 11:27

amd-argus Bot reviewed Oct 19, 2026

View reviewed changes

mkuznet1 force-pushed the ci/madengine-llama-70b-pp4 branch from 9030070 to 9cd0f3d Compare October 19, 2026 15:29

Copilot AI balanced review requested due to automatic review settings October 19, 2026 15:29

Copilot AI reviewed Oct 19, 2026

amd-argus Bot reviewed Oct 19, 2026

View reviewed changes

mkuznet1 force-pushed the ci/madengine-llama-70b-pp4 branch from 9cd0f3d to 72a3884 Compare October 20, 2026 10:49

amd-argus Bot reviewed Oct 20, 2026

View reviewed changes

mkuznet1 force-pushed the ci/madengine-llama-70b-pp4 branch from 72a3884 to 2f98664 Compare October 20, 2026 16:02

amd-argus Bot reviewed Oct 20, 2026

View reviewed changes

mkuznet1 added 3 commits October 22, 2026 11:19

Merge branch 'ci/madengine-maxtext-405b-stacked' into ci/madengine-ll…

b5e5f99

…ama-70b-pp4

Merge branch 'ci/madengine-maxtext-405b-stacked' into ci/madengine-ll…

af2957a

…ama-70b-pp4

Merge branch 'ci/madengine-maxtext-405b-stacked' into ci/madengine-ll…

58c0157

…ama-70b-pp4 Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # .github/workflows/therock-rccl-ci-linux.yml

amd-argus Bot reviewed Oct 22, 2026

View reviewed changes

Navigation Menu

Repository navigation

ci(rccl): add Llama-3.1-70B PP=4 to the weekly madengine runs - #13728

ci(rccl): add Llama-3.1-70B PP=4 to the weekly madengine runs#13728
mkuznet1 wants to merge 4 commits into
ci/madengine-maxtext-405b-stackedfrom
ci/madengine-llama-70b-pp4

mkuznet1 commented Oct 19, 2026 •

edited

Loading

Uh oh!

amd-argus Bot left a comment

Uh oh!

amd-argus Bot Oct 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

therock-pr-bot Bot commented Oct 19, 2026

Uh oh!

amd-argus Bot left a comment

Uh oh!

amd-argus Bot left a comment

Uh oh!

amd-argus Bot left a comment

Uh oh!

amd-argus Bot left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	# the runner account's quota.
	# RCCL prints this at WARN on every call of a deprecated API name
	# (src/misc/api_trace.cc), so MoE all-to-all fills the node logs with it.

Precision	tok/s/GPU	TFLOPS/GPU
BF16	1518.9	682.5
FP8	1658.7	745.4

Conversation

mkuznet1 commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Nightly vs weekly after this stack

Validation

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Findings

Uh oh!

amd-argus Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

therock-pr-bot Bot commented Oct 19, 2026

✅ All Policy Checks Passed

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mkuznet1 commented Oct 19, 2026 •

edited

Loading