ci(rccl): add DeepSeek-R1 SGLang 2P+2D to the weekly madengine runs

[mkuznet1]

Motivation

The madengine gate covers training only. This adds DeepSeek-R1 inference with
SGLang, prefill/decode disaggregated over 2 prefill and 2 decode nodes, so RCCL
carries the tensor-parallel collectives of an inference server under load. It
runs on the weekly schedule only.

Stacked on #13728 (Llama-3.1-70B PP=4), which is stacked on #13182.

Changes

• New workload deepseek-r1-sglang-disagg (family: sglang-disagg,
  type: inference, required_nodes: 4, 3 h time_limit).
  • Baseline: MAD's docker/sglang_disagg_inference_full_overlay recipe at
    MAD_REF, with its FROM $BASE_DOCKER pinned to the resolved base. The
    candidate is the same recipe plus the CI librccl swap, so the two phases
    differ only in RCCL. Both phases take the local-image path, as the
    maxtext baseline does: node 0 builds from the advertised recipe and the
    other nodes load its tar from WORK_DIR/docker_builds.
  • Gate metric: output_token_throughput_tok_s of the whole deployment at
    isl1024_osl1024_con64 and isl8192_osl1024_con64. Only those two
    points are run, three times each (BENCHMARK_ITR=3); MAD's parser keeps
    each point's best pass. A point is just 2 x concurrency requests, and a
    single pass at isl1024_osl1024_con64 moved 6.8% between two A/A phases
    on the same nodes.
  • OMP_NUM_THREADS=8 and the TORCH_NCCL_* settings are set in the
    workload's own env, as in the hand-run manifest this was validated with.
    The shared ones reach only the deployment env, not the container.
  • MOE_A2A_BACKEND=none: the MoE dispatch stays on RCCL, not MoRI.
  • The image's own rdma-core 63 is kept (mount_host_ib_libs: False); the
    hosts' rdma-core 61 lacks the bnxt_re QP fix.
  • Weights are mounted read-only from the new CLUSTER_CONFIGS key
    models_dir. run.sh downloads into MODEL_PATH when it finds the
    weights incomplete; read-only makes that fail instead of filling storage.
  • NCCL_DEBUG=INFO with NCCL_DEBUG_SUBSYS=INIT,NET: at WARN, RCCL prints
    its version banner only on rank 0 of a communicator, and the replacement
    check needs one per node.
• /run_logs is a per-phase directory, created 0777. Both phases share one
  SLURM_JOB_ID, which the launcher uses as its log directory name.
• The RCCL replacement check reads the sglang server logs
  (prefill_NODE*.log, decode_NODE*.log) when a phase has them.
• Results are paired by sweep point (_isl..._osl..._con... suffix of the
  row name) when a row has no precision.
• DNS preflight in the allocation: every node must resolve its own hostname to
  an address it has. A stale record otherwise leaves the sglang barrier
  waiting for that node until timeout. Nodes can be kept out with the
  repository variable RCCL_MADENGINE_SLURM_EXCLUDE (a SLURM hostlist, passed
  as MADENGINE_SLURM_EXCLUDE and validated before salloc --exclude).
• The upload step also keeps slurm_output/run_logs/*/*.log.
• Weekly matrix entry (nodes: '4', job_timeout_minutes: '360') and a
  dispatch choice.
• tests/test_sglang_disagg.py: manifest layout, recipe pin, result pairing,
  banner logs, DNS preflight and exclude handling, and that every gated point
  is in the benchmark sweep.
• CLUSTER_CONFIGS["ruby"]["models_dir"] is /apps/data/models; the weights
  are at /apps/data/models/DeepSeek-R1-0528, readable by the runner account.

Nightly vs weekly after this stack

Schedule	Workloads
Nightly (Sun-Fri cron)	llama-3.1-70b-training (FSDP2, 2N)
Weekly (Sat cron, test_scope=all), serial	gpt-oss-120b-training (2N), llama-3.1-70b-pp4-training (2N), llama-3.1-405b-maxtext (4N), deepseek-r1-sglang-disagg (4N)

Validation

• python3 -m unittest discover -s tests: 32 tests pass.

• A/A on ruby through this code path: generate_manifest, the in-place
  salloc with both preflights, madengine's sglang-disagg launcher, banner
  check and result pairing, with the baseline image in both phases (4 x 8
  MI355X). Output tok/s, phase 1 -> phase 2:

  SLURM job	Passes	isl1024_osl1024_con64	isl8192_osl1024_con64
  79505	1	1460.2 -> 1417.5 (-2.9%)	1154.2 -> 1152.9 (-0.1%)
  79615	1	1699.2 -> 1582.9 (-6.8%)	1223.2 -> 1211.2 (-1.0%)
  82589	best of 3	1513.9 -> 1527.2 (+0.9%)	1156.9 -> 1179.4 (+1.9%)

  79615 is why the gate runs three passes. Within a phase each pass is faster
  than the one before (82589 baseline, isl1024: 1348.6, 1410.5, 1513.9),
  which keeping the best pass absorbs. The absolute level depends on the node
  set (79505 and 82589 ran on the same four nodes), which is why both phases
  share one allocation. 82589 took 67 min end to end, image build included.

• CI dispatch of the new key: TODO, blocked by the runner account's disk
  quota on ruby.
  Run 39238800940
  failed with [Errno 122] Disk quota exceeded in "Create Python environment",
  before any allocation: the runner account's home was full. feat(rccl): score the madengine gate against a same-node baseline #12899 now keeps
  conda and pip out of that home.

[amd-argus]

ARGUS

Findings

5 more finding(s), not posted inline

• .github/workflows/therock-rccl-ci-linux.yml:286 (Low) Timeout comment attributes the extra 120 min to image builds, but builds run outside the script's 300-min timer
• projects/rccl/ci/scripts/test_madengine.py:762 (Low) rccl_banner_logs never checks that every node has a server log, so a node with no log is skipped in RCCL verification
• projects/rccl/ci/scripts/test_madengine.py:1557 (Low) Only the top run_logs dir is 0777; the launcher's nobody-owned subdir may block the runner's rm -rf cleanup
• projects/rccl/ci/scripts/tests/test_sglang_disagg.py:52 (Low) Test passes dockerfile_name/local_only but with no overlay_ctx file or registry they do not affect the manifest
• projects/rccl/ci/scripts/test_madengine.py:1368 (Low) sglang overlay recipe, install_madengine raises, skip-overlay refusal and PREFLIGHT_DNS content have no tests

Bottom line: Worth settling before approval: run_logs is created 0777 with no sticky bit (test_madengine.py:1557), so another cluster user can plant the logs the RCCL swap is verified from and that the new artifact glob uploads.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[amd-argus]

Medium

run_logs.chmod(0o777) has no sticky bit, and WORK_DIR sits under the world-writable /apps/rccl-ci. Any cluster user can rename the launcher's subdir and plant prefill_*/decode_* logs, which rccl_banner_logs (line 762) then trusts over the .out logs, and the new run_logs/*/*.log upload glob ships whatever is planted, symlinks included. Could we use 0o1777 and read and upload only the $SLURM_JOB_ID subdir, skipping symlinks?

[amd-argus]

Nit

Suggested change

	# The first inference workload: DeepSeek-R1 served disaggregated, two
	# DeepSeek-R1 served disaggregated, two

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	PR does not contain code files — Unit Test auto-passed
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?