feat(rccl): score the madengine gate against a same-node baseline by mkuznet1 · Pull Request #12899 · ROCm/rocm-systems

mkuznet1 · 2026-10-06T11:02:36Z

What this changes

The nightly MADEngine job scored one run against the rolling mean of the previous five. This replaces that with an A/B: the workload runs twice inside a single SLURM allocation — once on the stock base image, once on the same image
with the CI-built RCCL laid over it — and the gate is the ratio of the two.

Why

Each nightly lands on whichever nodes the scheduler picks, and the node pair, not RCCL, dominates the absolute throughput. Measured over four weeks of this job's own history:

	across node pairs	same pair, repeated
BF16	cv 2.18%, peak-to-peak 7.47%	0.02%
FP8	cv 2.70%, peak-to-peak 9.57%	0.25%

The repeat was three weeks apart on two different RCCL builds. So the workload is reproducible to a fraction of a percent on fixed hardware, and the 2% threshold sits below one standard deviation of the node-assignment noise.
Replaying the shipped gate over that history trips 3 of 11 scored runs on BF16 and 4 of 11 on FP8. AICOMRCCL-2456 was one of those false positives.

Comparing two runs on the same nodes divides the node term out. The ratio is not 1.0 and does not need to be — it needs to be stable, and a real RCCL regression moves it while an unlucky node pair does not.

Design notes

Why the baseline is the stock image. Not a free choice. The overlay is tagged {base_image}-rccl-{gpu_target}-{rccl_commit} and is never pushed anywhere, so it only exists inside the allocation that built it. "Yesterday's
overlay as baseline" is not implementable without standing up a registry first. The base image is pullable and its layers are already on the node from the overlay build, so the extra pull is cheap. What the gate then measures is the
CI RCCL against the RCCL shipping in the image, on identical hardware.

Why one allocation and not two jobs. A second job pinned to the first one's nodes is not schedulable until those exact nodes free up, reserves nothing while it waits and accrues no priority — so on a busy shared partition the comparison would intermittently not happen at all. A gate that sometimes produces no answer is worse than a noisy one. salloc holds the nodes across both runs; madengine picks up SLURM_JOB_ID and runs in place via srun instead of submitting its own sbatch.

Images are built on the compute nodes only. The runner host used to docker build the overlay under a per-run tag that nothing removed, and the nodes never used it: they rebuild the same tag from the manifest's Dockerfile (node 0 builds and docker saves to MAD_DOCKER_BUILDS on shared storage, the others docker load). All runners on that host share its Docker disk, which ran out of space (run 39407041026). Now the runner host only stages the Dockerfile and build context; --registry and the single-node pin to the build host are gone (the workflow never passed --registry). After both phases the run removes its built tag from every allocated node (ab/cleanup_images.sh; a run killed by the timeout leaves it). The tar on shared storage is deleted by the cleanup step, which also removes from the runner host the tags earlier versions built and the workloads' base images, without -f, so an image still in use stays.

Why each phase gets its own directory. madengine writes perf.csv, perf_entry_super.json and slurm_output/ under fixed names relative to its working directory, so a shared one would let the second run overwrite the first.

Datastore. Absolute values are still recorded, one row per phase per precision, now carrying phase, image, nodelist, slurm_job_id and, on the candidate, ratio. The node list is new: the analysis above had to recover it by joining perf_entry_super.json:machine_name against sacct.

Every expected precision is scored. A workload lists the precisions MAD trains ("precisions"; BF16 and FP8 for Llama-3.1-70B on gfx950). One missing from either phase, or from both, is a "no comparison" failure rather than a green run on the precisions that are left.

Phase timeout. Each phase's madengine timeout is the allocation's time_limit, so salloc --time is the only cap; a null card timeout would fall back to madengine's 2h default.

Cost. Two runs per nightly instead of one. time_limit goes 3h→4h (70B) and 2h→3h (scout), --timeout-minutes 190→330, and the workflow step 210→360 minutes.

Dependencies

madengine at ROCm/madengine develop 00ee3e8, which carries ROCm/madengine#213 (run a multi-node workload in place inside an existing allocation) and ROCm/madengine#190 (job-scoped node workspaces, stale-results cleanup). Without chore: unset executable permission #213 the inner madengine run would submit its own sbatch and both phases would leave the allocation.
MAD at the mad-rccl tip 7a9a25e.

Testing

End to end on ruby (gfx950, 2 nodes, meta64):

run 37494846163, Llama-3.1-70B: PASSED, BF16 ratio 1.0042, FP8 0.9982, both phases in one allocation (job 59796).
run 38238634668, GPT-OSS-120B from the follow-up feat(rccl): add GPT-OSS-120B on 2 nodes to the madengine gate #13180 on this branch: PASSED, BF16 0.9993, FP8 0.9924 (job 70409).

The in-place execution path was also validated on a live two-node allocation while developing madengine#213: two runs under one job id with distinct run ids, non-colliding per-node logs, and srun reaching both nodes.

JIRA ID AICOMNET-420

therock-pr-bot · 2026-10-06T11:03:38Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	PR does not contain code files — Unit Test auto-passed
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-10-06T11:03:41Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

coketaste

The same-node A/B is the right gate. Across this job's own history the node pair moves throughput by more than the 2% threshold, and a repeat on one pair stays well under it, so scoring candidate against baseline inside one allocation is what makes that threshold mean something. The stock image is the only baseline this workflow can pull, and giving each phase its own directory is required because madengine writes perf.csv and slurm_output/ under fixed names.

Not ready to leave draft:

MADENGINE_REPO / MADENGINE_REF still point at mkuznet1/madengine 10a0414. Until that is on ROCm/madengine, the inner madengine run submits its own sbatch and both phases leave the allocation.
This has not been run end to end. The 35 local checks are not in the tree, so CI cannot re-run them (the policy bot auto-passes unit tests on this script).

Two behaviors to fix or explicitly accept are noted inline.

Four fixes from review. The first two were fail-closed. Metric alias. extract_maxtext_perf.py writes `tok_per_s_per_gpu`, already an average of the last 10 steps, and madengine copies the name through. The global alias accepts only `tok_per_s_per_gpu_avg` after #12899, so every structured throughput row was filtered out and the live-log fallback only understands Megatron's format: a finished run produced no comparison and failed closed. Accepted per workload via a new `metric_aliases` key rather than widened globally -- Megatron emits both names with different meanings, and widening would score llama-3.1-70b on its final iteration again. The stale comment claiming the name was already in _METRIC_ALIASES was true before the rebase onto #12899 and is gone. Timeout. The script's budget covers queueing, because the allocation is taken inside it. At the 330 default a 5 h time_limit left 30 minutes of queue slack, against the 90 the 2-node workloads get; a dispatch of this job waited 13 h 14 m for a runner, so it would have been killed while still queued, reporting that neither phase ran. `timeout_minutes` is an input again, 360 by default so the matrix workloads are unchanged, 420 for this one -- 5 h plus the same 90 minutes. Node count. The workflow's `nodes` input defaults to 2, so a hand dispatch selecting this workload and leaving the rest alone took a 2-node exclusive allocation and OOMed in step 0. `required_nodes` is checked in main() before anything allocates. Registry. generate_manifest() marked any non-pull_only image as registry-backed when --registry was set, but build_maxtext_baseline_image() has nothing to push to, so nodes would pull a tag that was never uploaded instead of rebuilding from the advertised Dockerfile. A `local_only` flag keeps that phase on the local-image path. The weekly job passes no --registry, so this was latent. patchelf's RPATH-to-RUNPATH conversion is left as MAD#228 has it, per review -- the gate shows resolution is unaffected, and changing it without a run that shows the loaded libamd_smi moved would be an untested variation. Verified: the 405B phase now collects its row while llama-3.1-70b still ignores the un-suffixed name; all four workloads get >= 90 minutes of queue slack; the baseline stays local_image under --registry while the candidate does not; all six Megatron phase manifests remain byte-identical.

…e gate Adds `llama-3.1-405b-maxtext` at 4 nodes (32 GPUs) as the first JAX workload in the gate, and the first whose base image is not rocm/primus. Stacked on Draft until #12899 and #13180 land. The enabling work is ROCm/MAD#228, which validated this workload at 4N. Model family ------------ Until now every workload was PyTorch/Megatron-LM out of MAD's scripts/primus_megatron-lm. A "family" key now dispatches the three places that differ: install_madengine() copies scripts/jax-maxtext and checks Primus out, generate_manifest() emits `--config_path <yaml>` plus a `distributed.primus` block instead of `--model_repo`, and the overlay builder uses a different recipe. Settings lifted from MAD#228's template are each a failure mode, not taste: XLA_GPU_AUTOTUNE_LEVEL=0 Primus defaults to level 4, which at 32 ranks presents as a dead hang -- no steps, no NCCL output -- rather than a slow start. The MAD-native XLA_AUTOTUNE_LEVEL is a silent no-op. per_device_batch_size=1 the shipped config OOMs at 241.25 GiB for jit_train_step at 4 nodes. --ulimit nofile=1048576 a 32-device clique exhausts the default and stalls silently at "Initialize clique". They ride the per-workload env_vars key #13180 introduced. Why this family needs its own A/B baseline ------------------------------------------ The Megatron baseline is the stock registry image. That cannot work here: rocm/jax-training:maxtext-* ships MaxText but no Primus, so scripts/jax-maxtext/run.sh would not find primus-cli and the baseline phase would die before step 0. So the baseline is built: base image plus the Primus payload, stock RCCL. The candidate is that same Dockerfile plus the librccl swap -- literally a prefix, so the payload layers come out of the cache and only the swap builds twice. The pair therefore differs by the RCCL swap and nothing else, which is tighter than the Megatron A/B rather than looser. The baseline manifest advertises its own Dockerfile. Left at the default a compute node would rebuild the swapped image under the baseline tag, giving an A/B that compares the CI RCCL against itself. Overlay ------- The swap for this family is discovery-based -- find, per-target RUNPATH preservation, and an unresolved-soname set gate -- ported from MAD#228, rather than aimed at the Primus SDK venv paths, which this base lacks. The libamd_smi soname bridge is kept and now runs over every directory the sweep finds, ordered after the copy and before the post-swap measurement so a .27-against-.26 skew is reconciled instead of failing the gate. Unlike MAD#228's dockerfile the Primus payload is gated on primus-cli, not examples/run_pretrain.sh: AMD-AGI/Primus#999 retired that file and run.sh drives primus-cli. v26.7.0 still carries both so MAD's gate passes today, but it would hard-fail on jax-maxtext-v26.8. PRIMUS_REF is pinned here. --skip-overlay-build is refused for this family: its bind-mount targets are Primus-image paths absent from this base, and `docker run -v` creates a missing source as an empty directory rather than refusing, so the run would look healthy while measuring the image's own RCCL. Scheduling ---------- Its own job rather than a third matrix entry: the weekly gate has to be a job-level `if:`, where the matrix context is unavailable, and it is fixed at 4 nodes while the matrix entries take madengine_nodes. WORK_DIR already carries inputs.workload, so they do not collide. Not gated on the matrix job -- a 70B failure should not hide a 405B result. time_limit is 05:00:00 for the A/B pair, against MAD#228's 12:00:00 for a single run: a 4-node exclusive request with a 12 h limit backfills badly, the measured run is ~17 min of training (MAD#228 reports 17.40 s/step) plus prep, and 300 min stays inside the step's existing 330 so the script still reports. The existing 360/330 budget needed no change. Risk to the existing workloads ------------------------------ MAD_REF moves 07ecef61 -> 7acfac7c, which is exactly one commit, ROCm/MAD#279, repointing tools/fetch_primus.sh at a ref that still exists. It touches benchmark/jax_maxtext/README.md, docker/primus_maxdiffusion.ubuntu.amd.Dockerfile and tools/fetch_primus.sh -- nothing under scripts/primus_megatron-lm. I rendered both A/B phase manifests for llama-3.1-70b, llama-4-scout and gpt-oss-120b against this branch and its base: byte-identical, all six. Also verified: the 405B manifest matches MAD#228's validated template field for field; the baseline and candidate manifests differ only in image and dockerfile; both generated Dockerfiles parse under `bash -n`, with the bridge ordered after the copy and before the gate. Not yet run on hardware. The first run has to establish whether the TheRock kpack librccl loads on this base at all, and whether its libamd_smi soname needs the bridge here. JIRA ID AICOMNET-420

Four fixes from review. The first two were fail-closed. Metric alias. extract_maxtext_perf.py writes `tok_per_s_per_gpu`, already an average of the last 10 steps, and madengine copies the name through. The global alias accepts only `tok_per_s_per_gpu_avg` after #12899, so every structured throughput row was filtered out and the live-log fallback only understands Megatron's format: a finished run produced no comparison and failed closed. Accepted per workload via a new `metric_aliases` key rather than widened globally -- Megatron emits both names with different meanings, and widening would score llama-3.1-70b on its final iteration again. The stale comment claiming the name was already in _METRIC_ALIASES was true before the rebase onto #12899 and is gone. Timeout. The script's budget covers queueing, because the allocation is taken inside it. At the 330 default a 5 h time_limit left 30 minutes of queue slack, against the 90 the 2-node workloads get; a dispatch of this job waited 13 h 14 m for a runner, so it would have been killed while still queued, reporting that neither phase ran. `timeout_minutes` is an input again, 360 by default so the matrix workloads are unchanged, 420 for this one -- 5 h plus the same 90 minutes. Node count. The workflow's `nodes` input defaults to 2, so a hand dispatch selecting this workload and leaving the rest alone took a 2-node exclusive allocation and OOMed in step 0. `required_nodes` is checked in main() before anything allocates. Registry. generate_manifest() marked any non-pull_only image as registry-backed when --registry was set, but build_maxtext_baseline_image() has nothing to push to, so nodes would pull a tag that was never uploaded instead of rebuilding from the advertised Dockerfile. A `local_only` flag keeps that phase on the local-image path. The weekly job passes no --registry, so this was latent. patchelf's RPATH-to-RUNPATH conversion is left as MAD#228 has it, per review -- the gate shows resolution is unaffected, and changing it without a run that shows the loaded libamd_smi moved would be an untested variation. Verified: the 405B phase now collects its row while llama-3.1-70b still ignores the un-suffixed name; all four workloads get >= 90 minutes of queue slack; the baseline stays local_image under --registry while the candidate does not; all six Megatron phase manifests remain byte-identical.

mkuznet1 · 2026-10-14T10:20:54Z

Pins are on upstream now: ROCm/madengine develop 00ee3e8 (has #190 and #213) and MAD mad-rccl 7a9a25e. Ran end to end: run 37494846163 (Llama-3.1-70B) and run 38238634668 (GPT-OSS-120B via #13180), both PASSED. The CI runs replace the local checks. Moving out of draft.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

…e gate Adds `llama-3.1-405b-maxtext` at 4 nodes (32 GPUs) as the first JAX workload in the gate, and the first whose base image is not rocm/primus. Stacked on Draft until #12899 and #13180 land. The enabling work is ROCm/MAD#228, which validated this workload at 4N. Model family ------------ Until now every workload was PyTorch/Megatron-LM out of MAD's scripts/primus_megatron-lm. A "family" key now dispatches the three places that differ: install_madengine() copies scripts/jax-maxtext and checks Primus out, generate_manifest() emits `--config_path <yaml>` plus a `distributed.primus` block instead of `--model_repo`, and the overlay builder uses a different recipe. Settings lifted from MAD#228's template are each a failure mode, not taste: XLA_GPU_AUTOTUNE_LEVEL=0 Primus defaults to level 4, which at 32 ranks presents as a dead hang -- no steps, no NCCL output -- rather than a slow start. The MAD-native XLA_AUTOTUNE_LEVEL is a silent no-op. per_device_batch_size=1 the shipped config OOMs at 241.25 GiB for jit_train_step at 4 nodes. --ulimit nofile=1048576 a 32-device clique exhausts the default and stalls silently at "Initialize clique". They ride the per-workload env_vars key #13180 introduced. Why this family needs its own A/B baseline ------------------------------------------ The Megatron baseline is the stock registry image. That cannot work here: rocm/jax-training:maxtext-* ships MaxText but no Primus, so scripts/jax-maxtext/run.sh would not find primus-cli and the baseline phase would die before step 0. So the baseline is built: base image plus the Primus payload, stock RCCL. The candidate is that same Dockerfile plus the librccl swap -- literally a prefix, so the payload layers come out of the cache and only the swap builds twice. The pair therefore differs by the RCCL swap and nothing else, which is tighter than the Megatron A/B rather than looser. The baseline manifest advertises its own Dockerfile. Left at the default a compute node would rebuild the swapped image under the baseline tag, giving an A/B that compares the CI RCCL against itself. Overlay ------- The swap for this family is discovery-based -- find, per-target RUNPATH preservation, and an unresolved-soname set gate -- ported from MAD#228, rather than aimed at the Primus SDK venv paths, which this base lacks. The libamd_smi soname bridge is kept and now runs over every directory the sweep finds, ordered after the copy and before the post-swap measurement so a .27-against-.26 skew is reconciled instead of failing the gate. Unlike MAD#228's dockerfile the Primus payload is gated on primus-cli, not examples/run_pretrain.sh: AMD-AGI/Primus#999 retired that file and run.sh drives primus-cli. v26.7.0 still carries both so MAD's gate passes today, but it would hard-fail on jax-maxtext-v26.8. PRIMUS_REF is pinned here. --skip-overlay-build is refused for this family: its bind-mount targets are Primus-image paths absent from this base, and `docker run -v` creates a missing source as an empty directory rather than refusing, so the run would look healthy while measuring the image's own RCCL. Scheduling ---------- Its own job rather than a third matrix entry: the weekly gate has to be a job-level `if:`, where the matrix context is unavailable, and it is fixed at 4 nodes while the matrix entries take madengine_nodes. WORK_DIR already carries inputs.workload, so they do not collide. Not gated on the matrix job -- a 70B failure should not hide a 405B result. time_limit is 05:00:00 for the A/B pair, against MAD#228's 12:00:00 for a single run: a 4-node exclusive request with a 12 h limit backfills badly, the measured run is ~17 min of training (MAD#228 reports 17.40 s/step) plus prep, and 300 min stays inside the step's existing 330 so the script still reports. The existing 360/330 budget needed no change. Risk to the existing workloads ------------------------------ MAD_REF moves 07ecef61 -> 7acfac7c, which is exactly one commit, ROCm/MAD#279, repointing tools/fetch_primus.sh at a ref that still exists. It touches benchmark/jax_maxtext/README.md, docker/primus_maxdiffusion.ubuntu.amd.Dockerfile and tools/fetch_primus.sh -- nothing under scripts/primus_megatron-lm. I rendered both A/B phase manifests for llama-3.1-70b, llama-4-scout and gpt-oss-120b against this branch and its base: byte-identical, all six. Also verified: the 405B manifest matches MAD#228's validated template field for field; the baseline and candidate manifests differ only in image and dockerfile; both generated Dockerfiles parse under `bash -n`, with the bridge ordered after the copy and before the gate. Not yet run on hardware. The first run has to establish whether the TheRock kpack librccl loads on this base at all, and whether its libamd_smi soname needs the bridge here. JIRA ID AICOMNET-420

Four fixes from review. The first two were fail-closed. Metric alias. extract_maxtext_perf.py writes `tok_per_s_per_gpu`, already an average of the last 10 steps, and madengine copies the name through. The global alias accepts only `tok_per_s_per_gpu_avg` after #12899, so every structured throughput row was filtered out and the live-log fallback only understands Megatron's format: a finished run produced no comparison and failed closed. Accepted per workload via a new `metric_aliases` key rather than widened globally -- Megatron emits both names with different meanings, and widening would score llama-3.1-70b on its final iteration again. The stale comment claiming the name was already in _METRIC_ALIASES was true before the rebase onto #12899 and is gone. Timeout. The script's budget covers queueing, because the allocation is taken inside it. At the 330 default a 5 h time_limit left 30 minutes of queue slack, against the 90 the 2-node workloads get; a dispatch of this job waited 13 h 14 m for a runner, so it would have been killed while still queued, reporting that neither phase ran. `timeout_minutes` is an input again, 360 by default so the matrix workloads are unchanged, 420 for this one -- 5 h plus the same 90 minutes. Node count. The workflow's `nodes` input defaults to 2, so a hand dispatch selecting this workload and leaving the rest alone took a 2-node exclusive allocation and OOMed in step 0. `required_nodes` is checked in main() before anything allocates. Registry. generate_manifest() marked any non-pull_only image as registry-backed when --registry was set, but build_maxtext_baseline_image() has nothing to push to, so nodes would pull a tag that was never uploaded instead of rebuilding from the advertised Dockerfile. A `local_only` flag keeps that phase on the local-image path. The weekly job passes no --registry, so this was latent. patchelf's RPATH-to-RUNPATH conversion is left as MAD#228 has it, per review -- the gate shows resolution is unaffected, and changing it without a run that shows the loaded libamd_smi moved would be an untested variation. Verified: the 405B phase now collects its row while llama-3.1-70b still ignores the un-suffixed name; all four workloads get >= 90 minutes of queue slack; the baseline stays local_image under --registry while the candidate does not; all six Megatron phase manifests remain byte-identical.

mkuznet1 · 2026-10-14T15:11:12Z

@amd-argus review

amd-argus

ARGUS

Findings

3 more finding(s), not posted inline

projects/rccl/ci/scripts/test_madengine.py:1203 (Low) New gate logic (pair_by_precision, check_ab_regression) has no offline test; both E2E runs only hit the passing path
projects/rccl/ci/scripts/test_madengine.py:716 (Low) generate_manifest builds two image_entry dicts that repeat docker_image, build_status, build_duration and gpu_vendor
projects/rccl/ci/scripts/test_madengine.py:1584 (Low) phase_images re-states the phase-to-image mapping already in the phase loop at line 1453

This PR changes 1,110 lines across 2 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Looks approval ready.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

…e gate Adds `llama-3.1-405b-maxtext` at 4 nodes (32 GPUs) as the first JAX workload in the gate, and the first whose base image is not rocm/primus. Stacked on Draft until #12899 and #13180 land. The enabling work is ROCm/MAD#228, which validated this workload at 4N. Model family ------------ Until now every workload was PyTorch/Megatron-LM out of MAD's scripts/primus_megatron-lm. A "family" key now dispatches the three places that differ: install_madengine() copies scripts/jax-maxtext and checks Primus out, generate_manifest() emits `--config_path <yaml>` plus a `distributed.primus` block instead of `--model_repo`, and the overlay builder uses a different recipe. Settings lifted from MAD#228's template are each a failure mode, not taste: XLA_GPU_AUTOTUNE_LEVEL=0 Primus defaults to level 4, which at 32 ranks presents as a dead hang -- no steps, no NCCL output -- rather than a slow start. The MAD-native XLA_AUTOTUNE_LEVEL is a silent no-op. per_device_batch_size=1 the shipped config OOMs at 241.25 GiB for jit_train_step at 4 nodes. --ulimit nofile=1048576 a 32-device clique exhausts the default and stalls silently at "Initialize clique". They ride the per-workload env_vars key #13180 introduced. Why this family needs its own A/B baseline ------------------------------------------ The Megatron baseline is the stock registry image. That cannot work here: rocm/jax-training:maxtext-* ships MaxText but no Primus, so scripts/jax-maxtext/run.sh would not find primus-cli and the baseline phase would die before step 0. So the baseline is built: base image plus the Primus payload, stock RCCL. The candidate is that same Dockerfile plus the librccl swap -- literally a prefix, so the payload layers come out of the cache and only the swap builds twice. The pair therefore differs by the RCCL swap and nothing else, which is tighter than the Megatron A/B rather than looser. The baseline manifest advertises its own Dockerfile. Left at the default a compute node would rebuild the swapped image under the baseline tag, giving an A/B that compares the CI RCCL against itself. Overlay ------- The swap for this family is discovery-based -- find, per-target RUNPATH preservation, and an unresolved-soname set gate -- ported from MAD#228, rather than aimed at the Primus SDK venv paths, which this base lacks. The libamd_smi soname bridge is kept and now runs over every directory the sweep finds, ordered after the copy and before the post-swap measurement so a .27-against-.26 skew is reconciled instead of failing the gate. Unlike MAD#228's dockerfile the Primus payload is gated on primus-cli, not examples/run_pretrain.sh: AMD-AGI/Primus#999 retired that file and run.sh drives primus-cli. v26.7.0 still carries both so MAD's gate passes today, but it would hard-fail on jax-maxtext-v26.8. PRIMUS_REF is pinned here. --skip-overlay-build is refused for this family: its bind-mount targets are Primus-image paths absent from this base, and `docker run -v` creates a missing source as an empty directory rather than refusing, so the run would look healthy while measuring the image's own RCCL. Scheduling ---------- Its own job rather than a third matrix entry: the weekly gate has to be a job-level `if:`, where the matrix context is unavailable, and it is fixed at 4 nodes while the matrix entries take madengine_nodes. WORK_DIR already carries inputs.workload, so they do not collide. Not gated on the matrix job -- a 70B failure should not hide a 405B result. time_limit is 05:00:00 for the A/B pair, against MAD#228's 12:00:00 for a single run: a 4-node exclusive request with a 12 h limit backfills badly, the measured run is ~17 min of training (MAD#228 reports 17.40 s/step) plus prep, and 300 min stays inside the step's existing 330 so the script still reports. The existing 360/330 budget needed no change. Risk to the existing workloads ------------------------------ MAD_REF moves 07ecef61 -> 7acfac7c, which is exactly one commit, ROCm/MAD#279, repointing tools/fetch_primus.sh at a ref that still exists. It touches benchmark/jax_maxtext/README.md, docker/primus_maxdiffusion.ubuntu.amd.Dockerfile and tools/fetch_primus.sh -- nothing under scripts/primus_megatron-lm. I rendered both A/B phase manifests for llama-3.1-70b, llama-4-scout and gpt-oss-120b against this branch and its base: byte-identical, all six. Also verified: the 405B manifest matches MAD#228's validated template field for field; the baseline and candidate manifests differ only in image and dockerfile; both generated Dockerfiles parse under `bash -n`, with the bridge ordered after the copy and before the gate. Not yet run on hardware. The first run has to establish whether the TheRock kpack librccl loads on this base at all, and whether its libamd_smi soname needs the bridge here. JIRA ID AICOMNET-420

Four fixes from review. The first two were fail-closed. Metric alias. extract_maxtext_perf.py writes `tok_per_s_per_gpu`, already an average of the last 10 steps, and madengine copies the name through. The global alias accepts only `tok_per_s_per_gpu_avg` after #12899, so every structured throughput row was filtered out and the live-log fallback only understands Megatron's format: a finished run produced no comparison and failed closed. Accepted per workload via a new `metric_aliases` key rather than widened globally -- Megatron emits both names with different meanings, and widening would score llama-3.1-70b on its final iteration again. The stale comment claiming the name was already in _METRIC_ALIASES was true before the rebase onto #12899 and is gone. Timeout. The script's budget covers queueing, because the allocation is taken inside it. At the 330 default a 5 h time_limit left 30 minutes of queue slack, against the 90 the 2-node workloads get; a dispatch of this job waited 13 h 14 m for a runner, so it would have been killed while still queued, reporting that neither phase ran. `timeout_minutes` is an input again, 360 by default so the matrix workloads are unchanged, 420 for this one -- 5 h plus the same 90 minutes. Node count. The workflow's `nodes` input defaults to 2, so a hand dispatch selecting this workload and leaving the rest alone took a 2-node exclusive allocation and OOMed in step 0. `required_nodes` is checked in main() before anything allocates. Registry. generate_manifest() marked any non-pull_only image as registry-backed when --registry was set, but build_maxtext_baseline_image() has nothing to push to, so nodes would pull a tag that was never uploaded instead of rebuilding from the advertised Dockerfile. A `local_only` flag keeps that phase on the local-image path. The weekly job passes no --registry, so this was latent. patchelf's RPATH-to-RUNPATH conversion is left as MAD#228 has it, per review -- the gate shows resolution is unaffected, and changing it without a run that shows the loaded libamd_smi moved would be an untested variation. Verified: the 405B phase now collects its row while llama-3.1-70b still ignores the un-suffixed name; all four workloads get >= 90 minutes of queue slack; the baseline stays local_image under --registry while the candidate does not; all six Megatron phase manifests remain byte-identical.

amd-argus

ARGUS

This PR changes 1,090 lines across 2 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Worth settling before approval: on timeout, test_madengine.py:910 kills only salloc, so run_phases.sh and its madengine/srun children can keep running on the allocation.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

The stack conflicts when it lands in order, and this comment is why. #12899 reworded it to a one-liner after #13180 branched, so #13180 -- and this branch on top of it -- still carry the old two-line form. Landing #12899 first and then this branch makes both sides changed in the same region. Removing the `timeout-minutes` line from the step is substantive and stays. The explanation of why it is gone does not need to live in that hunk, so it moves up to the job-level timeout it is actually about, and the pre-existing comment is left exactly as #13180 has it. Only one side then touches those lines and the merge resolves on its own. The real staleness is that #13180 sits on an older ci/madengine-same-node-ab; rebasing it is that PR's call, not this one's. This just stops this branch from adding to the collision.

…ghput _METRIC_ALIASES matched tok_per_s_per_gpu, which madengine fills from the final logged iteration alone. That is a one-sample estimate: against a 50-iteration llama-3.1-70b reference it lands 1.8% low on roughly 3 of 48 iterations, a hair under the 2% regression gate, so a green build could fail on jitter. The tok_per_s_per_gpu_avg row carries Megatron's running average over the measured iterations and stays within 1% of the 50-iteration value from the third iteration onward, on both BF16 and FP8. It also removes a disagreement between the two parsing paths: parse_live_log_metrics() already reads the running average out of the live log, so the structured and fallback paths could score the same run differently. _TFLOPS_METRICS moves for the same reason, and because the field it feeds is already named tflops_avg. Baselines shift slightly, all well inside the gate: on llama-3.1-70b the measured difference is 0.04% on BF16 and 0.5% on FP8. Co-authored-by: Cursor <cursoragent@cursor.com>

The entry carried perf_primus-megatron-Llama-3.1-70B.csv, copied from the 70B workload it was derived from. MAD writes the scout results to perf_primus-megatron-Llama-4-Scout-17B-16E.csv, so the configured name either found nothing or, when both workloads had run into the same work directory, scored the scout job on the 70B numbers. Co-authored-by: Cursor <cursoragent@cursor.com>

The nightly compared one run against the rolling mean of the previous five. Those runs land wherever the scheduler puts them, and the node pair -- not RCCL -- dominates the absolute throughput: across four weeks of history it moves the number by more than the 2% gate, while a repeat on one and the same pair reproduced to a fraction of a percent on two different RCCL builds. The gate was therefore mostly reporting node assignment, and it fired on roughly a third of scored runs. Run the workload twice instead, inside a single allocation: once on the stock base image and once on the same image with the CI-built RCCL laid over it. Score the ratio. Node effects divide out, so the 2% threshold sits well clear of the noise rather than below it. The baseline has to be the stock image. The overlay is tagged per RCCL commit and, with no registry configured, never leaves the node that built it, so "yesterday's overlay" is not reachable. One allocation, not two jobs: a job pinned to another job's nodes is not schedulable until those nodes free up, reserves nothing while it waits and accrues no priority, so on a busy shared partition the comparison would intermittently not happen at all. Each phase gets its own working directory -- madengine writes perf.csv, perf_entry_super.json and slurm_output/ under fixed relative names, so sharing one would let the second run overwrite the first. Both absolute values and their ratio go to the datastore, now with the node list and job id attached, which the four-week analysis had to recover by hand. The madengine pin moves to a fork commit: running a multi-node workload in place inside an existing allocation is not on develop yet. Temporary, and marked as such in the source. Ref: AICOMNET-420 Co-authored-by: Cursor <cursoragent@cursor.com>

Picks up the in-place fixes the A/B depends on: the outer wait now honours the model card instead of capping each phase at 2h when no --timeout is passed, a failed phase no longer scancels the shared allocation, and a timed-out phase stops its srun step. Co-authored-by: Cursor <cursoragent@cursor.com>

Every compute node runs `madengine run` itself, from the environment it inherits through PATH. The venv sat on setup-python's 3.12, an Ubuntu build that needs glibc 2.35 and the runner's libssl, so it does not start on the nodes; a template patch papered over that by building a node-local venv and pip-installing madengine on every node of every job. Since the A/B moved the manifest into ab/<phase>/ that patch also looked for the madengine source in the wrong directory, and both phases of run 37466645356 died with "madengine source not found". - Replace setup-python + venv with a pinned, checksummed Miniforge under WORK_DIR and `conda create -p env python=3.12`. A conda-forge interpreter needs glibc 2.17 and carries its own libssl, so the same environment runs on the runner and on the nodes; the cleanup step still removes it. - Drop the per-node install patch. The stock template check now passes. - Drop patches that no longer do anything at the pinned madengine: $HOME/.local/bin is in the template already, NFS detection matches nfs[0-9]*, the multi-node SLURM_TMPDIR line carries the run suffix (so the anchor no longer matches), and run_orchestrator already wraps `yum info` in a timeout. - Replace the --gpus-per-node patch with madengine's skip_gpus_directive. Co-authored-by: Cursor <cursoragent@cursor.com>

…e summary Read the 'RCCL version :' banner each phase's node logs printed, so the report states which library was actually loaded on each side of the A/B (e.g. 2.28.9-HEAD:d34cbb6 from the stock image vs 2.32.3-develop:90d4836 from the CI build) instead of only the CI run tag. Co-authored-by: Cursor <cursoragent@cursor.com>

The A/B writes perf CSVs, manifests and slurm_output under WORK_DIR/ab/<phase>/, so the artifact paths at WORK_DIR root matched only madengine_summary.txt. Co-authored-by: Cursor <cursoragent@cursor.com>

madengine#213 is now stacked on #190, which job-scopes the single-node workspace under SLURM_TMPDIR and removes the node-local one after a successful task. Those are the two edits patch_madengine_for_cluster() made to job.sh.j2, so the patch goes. The new head also reads the job id out of run-tagged workspace names in the stale-workspace sweep. Co-authored-by: Cursor <cursoragent@cursor.com>

f7ef205 said the bumped pin makes the outer wait honour the model card instead of capping each phase at 2h. It does, but this manifest wrote "timeout": None, and madengine treats a null card timeout as unset, so every phase still got the 7200s default. Write the workload's time_limit in seconds instead, leaving the salloc --time as the only cap. Co-authored-by: Cursor <cursoragent@cursor.com>

madengine#190 and #213 are merged, so the in-place A/B no longer needs the fork: pin ROCm/madengine develop at 00ee3e8. It differs from the 9568510 the gate last ran on only by keeping an array task id intact in the workspace sweep and by a k8s test fix. Move MAD to the mad-rccl tip, 7a9a25e. Between the old pin and it, scripts/primus_megatron-lm changes only in its benchmark report and tests; the rest is Primus image bumps, GPT-OSS-120B on 2 nodes (#271), and jax/xdit fixes this gate does not run. Co-authored-by: Cursor <cursoragent@cursor.com>

…th phases pair_by_precision() built a row only for a precision that came back, so an FP8 leg missing from both phases produced no row and the gate stayed green on BF16. Seed the rows from the workload's "precisions" list, so a leg disabled upstream shows up as "no comparison" and fails the run. Llama-3.1-70B lists BF16 and FP8, which is what MAD's run.sh trains on gfx950. The datastore append now keys on whether any phase returned a row, so a run with nothing measured still records its fail entry. Co-authored-by: Cursor <cursoragent@cursor.com>

…store The per-run archive was runs/<run_id>/<phase>. Matrix legs share GITHUB_RUN_ID, so a second workload overwrote the first one's perf.csv and perf_entry_super.* there. Add the workload to that path. madengine_results.jsonl stays the single trend file. Those legs append to it in parallel on a shared filesystem, so take an exclusive flock around each write. Co-authored-by: Cursor <cursoragent@cursor.com>

Comments narrated the change ("until now", "This replaces", "now covers") or repeated the node-pair rationale that run_ab_in_one_allocation() already gives. Keep only what holds for the code as it stands. Co-authored-by: Cursor <cursoragent@cursor.com>

subprocess.run(timeout=) SIGKILLs only salloc. run_phases.sh and its madengine/srun children kept running on the allocation, while the workflow's cleanup removed env/ under them. Start salloc in its own session; on timeout scancel the job id from allocation.txt, then kill the whole process group. A request still queued has no job id yet, and killing salloc withdraws it. Checked with stub salloc/scancel/madengine and a 1-minute timeout: the old code left run_phases.sh and the hung madengine alive; this one cancels job 4242 and leaves no process behind. Co-authored-by: Cursor <cursoragent@cursor.com>

Without a "precisions" list the missing-precision guard had nothing to check for Scout. MAD's run.sh trains Llama-4-Scout-17B-16E on BF16 only on gfx950 (it falls to the default branch), so that is the list. Co-authored-by: Cursor <cursoragent@cursor.com>

…e gate Adds `llama-3.1-405b-maxtext` at 4 nodes (32 GPUs) as the first JAX workload in the gate, and the first whose base image is not rocm/primus. Stacked on Draft until #12899 and #13180 land. The enabling work is ROCm/MAD#228, which validated this workload at 4N. Model family ------------ Until now every workload was PyTorch/Megatron-LM out of MAD's scripts/primus_megatron-lm. A "family" key now dispatches the three places that differ: install_madengine() copies scripts/jax-maxtext and checks Primus out, generate_manifest() emits `--config_path <yaml>` plus a `distributed.primus` block instead of `--model_repo`, and the overlay builder uses a different recipe. Settings lifted from MAD#228's template are each a failure mode, not taste: XLA_GPU_AUTOTUNE_LEVEL=0 Primus defaults to level 4, which at 32 ranks presents as a dead hang -- no steps, no NCCL output -- rather than a slow start. The MAD-native XLA_AUTOTUNE_LEVEL is a silent no-op. per_device_batch_size=1 the shipped config OOMs at 241.25 GiB for jit_train_step at 4 nodes. --ulimit nofile=1048576 a 32-device clique exhausts the default and stalls silently at "Initialize clique". They ride the per-workload env_vars key #13180 introduced. Why this family needs its own A/B baseline ------------------------------------------ The Megatron baseline is the stock registry image. That cannot work here: rocm/jax-training:maxtext-* ships MaxText but no Primus, so scripts/jax-maxtext/run.sh would not find primus-cli and the baseline phase would die before step 0. So the baseline is built: base image plus the Primus payload, stock RCCL. The candidate is that same Dockerfile plus the librccl swap -- literally a prefix, so the payload layers come out of the cache and only the swap builds twice. The pair therefore differs by the RCCL swap and nothing else, which is tighter than the Megatron A/B rather than looser. The baseline manifest advertises its own Dockerfile. Left at the default a compute node would rebuild the swapped image under the baseline tag, giving an A/B that compares the CI RCCL against itself. Overlay ------- The swap for this family is discovery-based -- find, per-target RUNPATH preservation, and an unresolved-soname set gate -- ported from MAD#228, rather than aimed at the Primus SDK venv paths, which this base lacks. The libamd_smi soname bridge is kept and now runs over every directory the sweep finds, ordered after the copy and before the post-swap measurement so a .27-against-.26 skew is reconciled instead of failing the gate. Unlike MAD#228's dockerfile the Primus payload is gated on primus-cli, not examples/run_pretrain.sh: AMD-AGI/Primus#999 retired that file and run.sh drives primus-cli. v26.7.0 still carries both so MAD's gate passes today, but it would hard-fail on jax-maxtext-v26.8. PRIMUS_REF is pinned here. --skip-overlay-build is refused for this family: its bind-mount targets are Primus-image paths absent from this base, and `docker run -v` creates a missing source as an empty directory rather than refusing, so the run would look healthy while measuring the image's own RCCL. Scheduling ---------- Its own job rather than a third matrix entry: the weekly gate has to be a job-level `if:`, where the matrix context is unavailable, and it is fixed at 4 nodes while the matrix entries take madengine_nodes. WORK_DIR already carries inputs.workload, so they do not collide. Not gated on the matrix job -- a 70B failure should not hide a 405B result. time_limit is 05:00:00 for the A/B pair, against MAD#228's 12:00:00 for a single run: a 4-node exclusive request with a 12 h limit backfills badly, the measured run is ~17 min of training (MAD#228 reports 17.40 s/step) plus prep, and 300 min stays inside the step's existing 330 so the script still reports. The existing 360/330 budget needed no change. Risk to the existing workloads ------------------------------ MAD_REF moves 07ecef61 -> 7acfac7c, which is exactly one commit, ROCm/MAD#279, repointing tools/fetch_primus.sh at a ref that still exists. It touches benchmark/jax_maxtext/README.md, docker/primus_maxdiffusion.ubuntu.amd.Dockerfile and tools/fetch_primus.sh -- nothing under scripts/primus_megatron-lm. I rendered both A/B phase manifests for llama-3.1-70b, llama-4-scout and gpt-oss-120b against this branch and its base: byte-identical, all six. Also verified: the 405B manifest matches MAD#228's validated template field for field; the baseline and candidate manifests differ only in image and dockerfile; both generated Dockerfiles parse under `bash -n`, with the bridge ordered after the copy and before the gate. Not yet run on hardware. The first run has to establish whether the TheRock kpack librccl loads on this base at all, and whether its libamd_smi soname needs the bridge here. JIRA ID AICOMNET-420

Four fixes from review. The first two were fail-closed. Metric alias. extract_maxtext_perf.py writes `tok_per_s_per_gpu`, already an average of the last 10 steps, and madengine copies the name through. The global alias accepts only `tok_per_s_per_gpu_avg` after #12899, so every structured throughput row was filtered out and the live-log fallback only understands Megatron's format: a finished run produced no comparison and failed closed. Accepted per workload via a new `metric_aliases` key rather than widened globally -- Megatron emits both names with different meanings, and widening would score llama-3.1-70b on its final iteration again. The stale comment claiming the name was already in _METRIC_ALIASES was true before the rebase onto #12899 and is gone. Timeout. The script's budget covers queueing, because the allocation is taken inside it. At the 330 default a 5 h time_limit left 30 minutes of queue slack, against the 90 the 2-node workloads get; a dispatch of this job waited 13 h 14 m for a runner, so it would have been killed while still queued, reporting that neither phase ran. `timeout_minutes` is an input again, 360 by default so the matrix workloads are unchanged, 420 for this one -- 5 h plus the same 90 minutes. Node count. The workflow's `nodes` input defaults to 2, so a hand dispatch selecting this workload and leaving the rest alone took a 2-node exclusive allocation and OOMed in step 0. `required_nodes` is checked in main() before anything allocates. Registry. generate_manifest() marked any non-pull_only image as registry-backed when --registry was set, but build_maxtext_baseline_image() has nothing to push to, so nodes would pull a tag that was never uploaded instead of rebuilding from the advertised Dockerfile. A `local_only` flag keeps that phase on the local-image path. The weekly job passes no --registry, so this was latent. patchelf's RPATH-to-RUNPATH conversion is left as MAD#228 has it, per review -- the gate shows resolution is unaffected, and changing it without a run that shows the loaded libamd_smi moved would be an untested variation. Verified: the 405B phase now collects its row while llama-3.1-70b still ignores the un-suffixed name; all four workloads get >= 90 minutes of queue slack; the baseline stays local_image under --registry while the candidate does not; all six Megatron phase manifests remain byte-identical.

The stack conflicts when it lands in order, and this comment is why. #12899 reworded it to a one-liner after #13180 branched, so #13180 -- and this branch on top of it -- still carry the old two-line form. Landing #12899 first and then this branch makes both sides changed in the same region. Removing the `timeout-minutes` line from the step is substantive and stays. The explanation of why it is gone does not need to live in that hunk, so it moves up to the job-level timeout it is actually about, and the pre-existing comment is left exactly as #13180 has it. Only one side then touches those lines and the merge resolves on its own. The real staleness is that #13180 sits on an older ci/madengine-same-node-ab; rebasing it is that PR's call, not this one's. This just stops this branch from adding to the collision.

Runs 39238797725 and 39238800940 failed "Create Python environment" with "[Errno 122] Disk quota exceeded" while the work directory's filesystem had terabytes free: the quota that ran out is the runner account's home, which every other job of that account fills too. This job writes almost nothing there, but conda cannot finish without writing something: the Miniforge installer creates ~/.conda, conda create appends the new env to ~/.conda/environments.txt (and rolls the transaction back when that fails), and both cache notices under ~/.cache. pip caches every wheel it fetches under ~/.cache/pip. Point HOME into the conda prefix for that step only, and PIP_CACHE_DIR there for the rest of the job. The cleanup step already deletes the prefix. HOME is not exported to later steps, so whatever reaches the compute nodes still sees the real one. Checked by running this step's body, a pip install and the cleanup step with a read-only HOME: the old step fails at mkdir ~/.conda, the new one passes, HOME stays empty and nothing is left in WORK_DIR. Co-authored-by: Cursor <cursoragent@cursor.com>

…e gate Adds `llama-3.1-405b-maxtext` at 4 nodes (32 GPUs) as the first JAX workload in the gate, and the first whose base image is not rocm/primus. Stacked on Draft until #12899 and #13180 land. The enabling work is ROCm/MAD#228, which validated this workload at 4N. Model family ------------ Until now every workload was PyTorch/Megatron-LM out of MAD's scripts/primus_megatron-lm. A "family" key now dispatches the three places that differ: install_madengine() copies scripts/jax-maxtext and checks Primus out, generate_manifest() emits `--config_path <yaml>` plus a `distributed.primus` block instead of `--model_repo`, and the overlay builder uses a different recipe. Settings lifted from MAD#228's template are each a failure mode, not taste: XLA_GPU_AUTOTUNE_LEVEL=0 Primus defaults to level 4, which at 32 ranks presents as a dead hang -- no steps, no NCCL output -- rather than a slow start. The MAD-native XLA_AUTOTUNE_LEVEL is a silent no-op. per_device_batch_size=1 the shipped config OOMs at 241.25 GiB for jit_train_step at 4 nodes. --ulimit nofile=1048576 a 32-device clique exhausts the default and stalls silently at "Initialize clique". They ride the per-workload env_vars key #13180 introduced. Why this family needs its own A/B baseline ------------------------------------------ The Megatron baseline is the stock registry image. That cannot work here: rocm/jax-training:maxtext-* ships MaxText but no Primus, so scripts/jax-maxtext/run.sh would not find primus-cli and the baseline phase would die before step 0. So the baseline is built: base image plus the Primus payload, stock RCCL. The candidate is that same Dockerfile plus the librccl swap -- literally a prefix, so the payload layers come out of the cache and only the swap builds twice. The pair therefore differs by the RCCL swap and nothing else, which is tighter than the Megatron A/B rather than looser. The baseline manifest advertises its own Dockerfile. Left at the default a compute node would rebuild the swapped image under the baseline tag, giving an A/B that compares the CI RCCL against itself. Overlay ------- The swap for this family is discovery-based -- find, per-target RUNPATH preservation, and an unresolved-soname set gate -- ported from MAD#228, rather than aimed at the Primus SDK venv paths, which this base lacks. The libamd_smi soname bridge is kept and now runs over every directory the sweep finds, ordered after the copy and before the post-swap measurement so a .27-against-.26 skew is reconciled instead of failing the gate. Unlike MAD#228's dockerfile the Primus payload is gated on primus-cli, not examples/run_pretrain.sh: AMD-AGI/Primus#999 retired that file and run.sh drives primus-cli. v26.7.0 still carries both so MAD's gate passes today, but it would hard-fail on jax-maxtext-v26.8. PRIMUS_REF is pinned here. --skip-overlay-build is refused for this family: its bind-mount targets are Primus-image paths absent from this base, and `docker run -v` creates a missing source as an empty directory rather than refusing, so the run would look healthy while measuring the image's own RCCL. Scheduling ---------- Its own job rather than a third matrix entry: the weekly gate has to be a job-level `if:`, where the matrix context is unavailable, and it is fixed at 4 nodes while the matrix entries take madengine_nodes. WORK_DIR already carries inputs.workload, so they do not collide. Not gated on the matrix job -- a 70B failure should not hide a 405B result. time_limit is 05:00:00 for the A/B pair, against MAD#228's 12:00:00 for a single run: a 4-node exclusive request with a 12 h limit backfills badly, the measured run is ~17 min of training (MAD#228 reports 17.40 s/step) plus prep, and 300 min stays inside the step's existing 330 so the script still reports. The existing 360/330 budget needed no change. Risk to the existing workloads ------------------------------ MAD_REF moves 07ecef61 -> 7acfac7c, which is exactly one commit, ROCm/MAD#279, repointing tools/fetch_primus.sh at a ref that still exists. It touches benchmark/jax_maxtext/README.md, docker/primus_maxdiffusion.ubuntu.amd.Dockerfile and tools/fetch_primus.sh -- nothing under scripts/primus_megatron-lm. I rendered both A/B phase manifests for llama-3.1-70b, llama-4-scout and gpt-oss-120b against this branch and its base: byte-identical, all six. Also verified: the 405B manifest matches MAD#228's validated template field for field; the baseline and candidate manifests differ only in image and dockerfile; both generated Dockerfiles parse under `bash -n`, with the bridge ordered after the copy and before the gate. Not yet run on hardware. The first run has to establish whether the TheRock kpack librccl loads on this base at all, and whether its libamd_smi soname needs the bridge here. JIRA ID AICOMNET-420

Four fixes from review. The first two were fail-closed. Metric alias. extract_maxtext_perf.py writes `tok_per_s_per_gpu`, already an average of the last 10 steps, and madengine copies the name through. The global alias accepts only `tok_per_s_per_gpu_avg` after #12899, so every structured throughput row was filtered out and the live-log fallback only understands Megatron's format: a finished run produced no comparison and failed closed. Accepted per workload via a new `metric_aliases` key rather than widened globally -- Megatron emits both names with different meanings, and widening would score llama-3.1-70b on its final iteration again. The stale comment claiming the name was already in _METRIC_ALIASES was true before the rebase onto #12899 and is gone. Timeout. The script's budget covers queueing, because the allocation is taken inside it. At the 330 default a 5 h time_limit left 30 minutes of queue slack, against the 90 the 2-node workloads get; a dispatch of this job waited 13 h 14 m for a runner, so it would have been killed while still queued, reporting that neither phase ran. `timeout_minutes` is an input again, 360 by default so the matrix workloads are unchanged, 420 for this one -- 5 h plus the same 90 minutes. Node count. The workflow's `nodes` input defaults to 2, so a hand dispatch selecting this workload and leaving the rest alone took a 2-node exclusive allocation and OOMed in step 0. `required_nodes` is checked in main() before anything allocates. Registry. generate_manifest() marked any non-pull_only image as registry-backed when --registry was set, but build_maxtext_baseline_image() has nothing to push to, so nodes would pull a tag that was never uploaded instead of rebuilding from the advertised Dockerfile. A `local_only` flag keeps that phase on the local-image path. The weekly job passes no --registry, so this was latent. patchelf's RPATH-to-RUNPATH conversion is left as MAD#228 has it, per review -- the gate shows resolution is unaffected, and changing it without a run that shows the loaded libamd_smi moved would be an untested variation. Verified: the 405B phase now collects its row while llama-3.1-70b still ignores the un-suffixed name; all four workloads get >= 90 minutes of queue slack; the baseline stays local_image under --registry while the candidate does not; all six Megatron phase manifests remain byte-identical.

The stack conflicts when it lands in order, and this comment is why. #12899 reworded it to a one-liner after #13180 branched, so #13180 -- and this branch on top of it -- still carry the old two-line form. Landing #12899 first and then this branch makes both sides changed in the same region. Removing the `timeout-minutes` line from the step is substantive and stays. The explanation of why it is gone does not need to live in that hunk, so it moves up to the job-level timeout it is actually about, and the pre-existing comment is left exactly as #13180 has it. Only one side then touches those lines and the merge resolves on its own. The real staleness is that #13180 sits on an older ci/madengine-same-node-ab; rebasing it is that PR's call, not this one's. This just stops this branch from adding to the collision.

amd-argus

ARGUS

This PR changes 1,114 lines across 2 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

i-kosarev

some comments

i-kosarev · 2026-10-20T11:30:26Z

+    phase_dirs = dict(phases)
+
+    # Step 5: Run both phases on the same nodes, in one allocation
+    allocation = run_ab_in_one_allocation(


we should reject one-node runs: pinned madengine submits each phase as a separate sbatch, breaking same-node comparison.

i-kosarev · 2026-10-20T11:31:23Z

    if rccl_fingerprint.get("version"):
        rccl_ok, rccl_msg = verify_rccl_replacement(
-            work_dir, rccl_fingerprint,
+            phase_dirs[CANDIDATE], rccl_fingerprint,


we should require logs from every allocated node: verifying only discovered logs can accept an unverified candidate node e.g. if some of the nodes failed to produce logs

i-kosarev · 2026-10-20T11:35:58Z

+            if not perf_val:
+                continue
+            try:
+                val = float(perf_val)


we should reject non-finite or non-positive metrics before pairing; NaN currently passes comparison and can clear a failed phase

i-kosarev · 2026-10-20T11:36:31Z

-          rm -rf "${WORK_DIR}/madengine" "${WORK_DIR}/MAD" "${WORK_DIR}/venv" \
-                 "${WORK_DIR}/node_venv" "${WORK_DIR}/overlay_ctx" \
+          rm -rf "${WORK_DIR}/madengine" "${WORK_DIR}/MAD" "${WORK_DIR}/conda" \
+                 "${WORK_DIR}/env" "${WORK_DIR}/overlay_ctx" \


please also remove WORK_DIR/ab after upload; unique run directories otherwise retain large phase logs indefinitely

i-kosarev · 2026-10-20T11:37:04Z

+                    scale,
+                    row.get("metric_value"),
+                    row["status"],
+                    rccl_commit,


we should record the artifact run and content hash here; GITHUB_RUN_ID identifies the test, not the RCCL binary

i-kosarev · 2026-10-20T11:37:47Z

+# (ROCm/madengine#213), so the pin must not predate it.
 MADENGINE_REPO = "https://github.com/ROCm/madengine.git"
-MADENGINE_REF = "98217cd7ba721f5a5f2a8fb4729120cce1d57eac"  # v2.2.0, 2026-09-08
+MADENGINE_REF = "00ee3e80c0a308599633caf74e5d91a72164a2fe"  # develop, has #190 and #213


This pin leaves root-owned node workspaces behind: should we run container outputs under the runner UID?

Conflicts resolved toward this branch: it replaces setup-python with a conda env and drops patch_madengine_for_cluster for the pinned madengine. Kept develop's TheRock pin and PIP_NO_CACHE_DIR. Co-authored-by: Cursor <cursoragent@cursor.com>

The runner host built the candidate overlay under a per-run tag that was never removed, and the nodes never used it: with no registry they rebuild the same tag from the manifest's Dockerfile (node 0 builds and saves, the others load). Every runner on that host shares its Docker disk. - Only stage the overlay's Dockerfile and build context on the runner host; drop --registry and the single-node pin to the build host. - Remove this run's built tag from every allocated node after the phases. - Clean up remaining CI tags and the workloads' base images on the runner host, without -f so an image still in use stays. Co-authored-by: Cursor <cursoragent@cursor.com>

amd-argus

ARGUS

This PR changes 1,223 lines across 2 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

github-actions Bot added github actions Pull requests that update GitHub Actions code project: rccl labels Oct 6, 2026

therock-pr-bot Bot added the Not ready to Review label Oct 6, 2026

systems-assistant Bot added the organization: ROCm label Oct 6, 2026

mkuznet1 mentioned this pull request Oct 9, 2026

feat(rccl): add GPT-OSS-120B on 2 nodes to the madengine gate #13180

Open

i-kosarev added this pull request to stack #13181 October 9, 2026 11:49

i-kosarev mentioned this pull request Oct 9, 2026

feat(rccl): add JAX/MaxText Llama-3.1-405B on 4 nodes to the madengine gate #13182

Open

1 task done

coketaste reviewed Oct 9, 2026

View reviewed changes

Comment thread projects/rccl/ci/scripts/test_madengine.py

Comment thread projects/rccl/ci/scripts/test_madengine.py Outdated

mkuznet1 marked this pull request as ready for review October 14, 2026 10:21

mkuznet1 requested review from a team and a balanced review from Copilot October 14, 2026 10:21

Copilot AI reviewed Oct 14, 2026

mkuznet1 removed the Not ready to Review label Oct 14, 2026

mkuznet1 marked this pull request as draft October 14, 2026 15:09

mkuznet1 marked this pull request as ready for review October 14, 2026 15:09

amd-argus Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread projects/rccl/ci/scripts/test_madengine.py Outdated

amd-argus Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread projects/rccl/ci/scripts/test_madengine.py Outdated

mkuznet1 and others added 15 commits October 19, 2026 13:52

ci(rccl): upload the madengine results from each A/B phase directory

14fce83

The A/B writes perf CSVs, manifests and slurm_output under WORK_DIR/ab/<phase>/, so the artifact paths at WORK_DIR root matched only madengine_summary.txt. Co-authored-by: Cursor <cursoragent@cursor.com>

i-kosarev force-pushed the ci/madengine-same-node-ab branch from 634db0c to 50e9ea2 Compare October 19, 2026 13:52

mkuznet1 mentioned this pull request Oct 19, 2026

ci(rccl): add DeepSeek-R1 SGLang 2P+2D to the weekly madengine runs #13729

Open

amd-argus Bot reviewed Oct 19, 2026

View reviewed changes

i-kosarev reviewed Oct 20, 2026

View reviewed changes

mkuznet1 and others added 2 commits October 20, 2026 15:55

amd-argus Bot reviewed Oct 20, 2026

View reviewed changes

Conversation

mkuznet1 commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Why

Design notes

Dependencies

Testing

Uh oh!

therock-pr-bot Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Policy Checks Passed

Uh oh!

therock-pr-bot Bot commented Oct 6, 2026

Uh oh!

coketaste left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mkuznet1 commented Oct 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

mkuznet1 commented Oct 14, 2026

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Findings

Uh oh!

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

i-kosarev left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

i-kosarev Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

i-kosarev Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

i-kosarev Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

i-kosarev Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

i-kosarev Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

i-kosarev Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

amd-argus Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mkuznet1 commented Oct 6, 2026 •

edited

Loading

therock-pr-bot Bot commented Oct 6, 2026 •

edited

Loading

i-kosarev left a comment •

edited

Loading