Repository navigation
[https://nvbugs/6812381][fix] Split GB200-4_GPUs-PyTorch-Post-Merge into two shards - #20780
Conversation
…nto two shards
GB200-4_GPUs-PyTorch-Post-Merge-1 is the only shard (split 1/1) for the
42 post-merge cases of l0_gb200_multi_gpus. Those cases now need about
260 minutes of pytest time on one GB200 x4 node, but the Slurm walltime
is 240 minutes. Since 2026-09-02 the stage hit Slurm TIMEOUT in 75 of 80
post-merge runs. The Jenkins agent then drops ("The channel is closing
down"), and the cases after the cutoff never run.
Split the stage into two shards, as was done for the pre-merge
GB200-4_GPUs-PyTorch stages in NVIDIA#14836. The test list, the waivers and
the per-test timeouts do not change, and each case still runs in exactly
one shard.
Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
WalkthroughThe GB200 4-GPU post-merge SLURM configuration now runs two shards instead of one. Both shards use the same platform, test list, hardware, and execution settings. ChangesGB200 Test Sharding
Priority: ⬇️ Low Estimated code review effort: 1 (Trivial) | ~4 minutes Change: Bug fix Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation |
|---|---|---|
| Title check |  |
The title clearly identifies the fix and the affected GB200 post-merge stage. It follows the required NVBugs and type format. |
| Description check | |
The description explains the timeout issue, the two-shard solution, preserved test behavior, and test coverage with concrete results. It omits the PR Checklist section, but the required technical info… |
| Docstring Coverage | |
No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0… |
| Linked Issues check | |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
Full details: Description check
-
Explanation
The description explains the timeout issue, the two-shard solution, preserved test behavior, and test coverage with concrete results. It omits the PR Checklist section, but the required technical information is otherwise complete.
Full details: Docstring Coverage
-
Explanation
No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)
✨ Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
|
Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements. |
|
/bot run --stage-list "GB200-4_GPUs-PyTorch-Post-Merge-*" |
|
PR_Github #79829 [ run ] triggered by Bot. Commit: |
|
PR_Github #79829 [ run ] completed with state |
|
Bug engineer ownership @HuiGao-NV, as the engineer assigned to the linked bug, you are responsible for driving this bot PR to resolution. Please coordinate review, address feedback and required code changes, get CI passing, and verify the original failure before driving it to merge. If the PR is obsolete or superseded, close it with a reason. Keep this PR updated with blockers and the next step. |
|
/bot run --stage-list "GB200-4_GPUs-PyTorch-Post-Merge-*" |
|
PR_Github #79900 [ run ] triggered by Bot. Commit: |
|
PR_Github #79900 [ run ] completed with state |
|
/bot skip --comment "Splitted stages passed" |
|
PR_Github #79992 [ skip ] triggered by Bot. Commit: |
|
PR_Github #79992 [ skip ] completed with state |
Description
GB200-4_GPUs-PyTorch-Post-Merge-1is the only shard (split 1/1) for thestage: post_mergeblock ofl0_gb200_multi_gpus. It runs all 42 of those cases on one GB200 x4 node. Together they now need about 260 minutes of pytest time, but the Slurm walltime is 240 minutes. Since 2026-09-02 the stage hit SlurmTIMEOUTin 75 of 80 post-merge runs. The Jenkins agent then drops ("The channel is closing down"), and the cases after the cutoff never run.The stage did not get slower per case: the 27 cases that ran both in a passing August build and in a current timed-out build took 136 and 137 minutes. What changed is that several long cases were added or unwaived (the two Mistral-Large-3 cases,
TestNemotronV3Ultra::test_nvfp4_parallelism[ADP2_PP2], and several DeepSeek-V3-Lite PP cases). That pushed a stage that already ran at 220-230 minutes past the walltime.Fix
Split the stage into two shards,
GB200-4_GPUs-PyTorch-Post-Merge-1andGB200-4_GPUs-PyTorch-Post-Merge-2(split 1/2 and 2/2), as was done for the pre-mergeGB200-4_GPUs-PyTorch-*stages in #14836. The test list, the waivers and the per-test timeouts do not change, and each case still runs in exactly one shard.Test Coverage
The stage was run the way CI runs it (same image, same
--test-list, waivers, 3600 s per-test timeout,--splitting-algorithm least_durationwith the stage's--splits/--group), on GB200 x4 nodes with a 4-hour walltime, at 522daa8 with its post-merge wheel:TIMEOUTat 4:00:23. 38 passed, 2 skipped (waived), 0 failed;TestQwen3_235B_A22B::test_nvfp4_4gpus[latency_moe_trtllm_eagle3]was cut off mid-run andTestMoEComm::test_moe_comm_boundarynever started.The two shards together ran all 42 listed cases, each exactly once. Even if the split were completely unbalanced, the 21 longest cases add up to 201 minutes, which is still under the walltime.
scripts/test_to_stage_mapping.py --stages GB200-4_GPUs-PyTorch-Post-Merge-1 GB200-4_GPUs-PyTorch-Post-Merge-2returns the same 42 cases as the single stage did before.Fixes https://nvbugs/6812381
Dev Engineer Review
jenkins/L0_Test.groovyreplaces the single post-merge stage with two stages.l0_gb200_multi_gpus, four GPUs, and two-way splitting.QA Engineer Review
No test changes.
Per-File QA Perspective
jenkins/L0_Test.groovy: Verify that the two shards cover all 42 post-merge cases exactly once and retain the existing waivers and timeout settings.