ci: restore the image's cuDNN pin after reconciling requirements - #1836
Open
yueming-yuan wants to merge 3 commits into
Open
ci: restore the image's cuDNN pin after reconciling requirements#1836yueming-yuan wants to merge 3 commits into
yueming-yuan wants to merge 3 commits into
Conversation
The GPU jobs install requirements.txt inside the test container. torchft-nightly pulls torch, torch pins nvidia-cudnn-cu13 exactly, and that resolve uninstalls the cuDNN the image deliberately installed last: Collecting nvidia-cudnn-cu13==9.19.0.56 (from torch>=2.7->torchft-nightly->-r requirements.txt) Uninstalling nvidia-cudnn-cu13-9.22.0.52 Successfully installed nvidia-cudnn-cu13-9.19.0.56 transformer_engine 2.17's fused attention backward then fails with CUDNN_STATUS_BAD_PARAM on CUDNN_ATTR_OPERATION_RESHAPE_MODE, which is what has been failing test_lora_qwen2.5_0.5B, test_gpt_oss_20b_moe_lora_ci and test_qwen3_5_35b_a3b_lora_ci on #1795. Record the version the image shipped and reinstall it after the resolve, rather than hardcoding it here: the Dockerfile stays the single place the version is chosen, and --no-deps keeps the rest of the resolved set untouched. Declaring the pin in requirements.txt instead does not work -- pip resolves the conflict against torch's exact pin by downgrading torch itself to 2.10.0 and swapping the whole CUDA stack to cu12. Verified on an H200 devbox running the exact CI image digest: the requirements resolve reproduces the downgrade (9.22.0.52 -> 9.19.0.56, torch untouched), the restore puts 9.22.0.52 back, and TE fused attention backward then passes 8/8 configs that fail at 9.19.0.56.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
yueming-yuan
added a commit
that referenced
this pull request
Jul 28, 2026
Root cause found: the CI job's requirements resolve downgraded the image's cuDNN. Fixed in #1836, cherry-picked below.
GitHub runs 'shell: bash' as 'bash -e {0}'. '[ -n "$v" ] && pinned=...' as the last
statement of the loop body makes the loop exit non-zero whenever a package is absent
(cu12 on a cu13 image), which aborted the whole step and every GPU job with it. Use an
if-block so the loop always ends cleanly.
Verified by extracting the step from the workflow and running it under 'bash -e' with a
stub pip: exits 0 both when the version is untouched and when it has to be restored.
GitHub runs steps as 'bash --noprofile --norc -e -o pipefail'. 'pip show' exits 1 for a package that is not installed (cu12 on a cu13 image), pipefail propagates that through the awk pipe, and -e then aborts the step -- taking every GPU job with it. Terminate both command substitutions with '|| true', matching how this file already reads optional values out of the PR body. Verified by extracting the step from the workflow and running it under the exact shell line GitHub logs, with a stub pip where cu12 is absent: exits 0, and still restores the pin when the resolve moves it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What breaks
Every GPU job runs
pip install -r requirements.txtinside the test container("Reconcile Miles dependencies").
torchft-nightlypullstorch,torchpinsnvidia-cudnn-cu13exactly, and the resolve therefore uninstalls the cuDNN theimage deliberately installed last:
transformer_engine2.17's fused attention backward then fails:This is what has been failing
test_lora_qwen2.5_0.5B,test_gpt_oss_20b_moe_lora_ciandtest_qwen3_5_35b_a3b_lora_cion#1795. The Dockerfile's pin was never wrong — it was being undone at
test time, which is why moving it had no effect.
How this fixes it
Record the cuDNN version the image shipped, run the resolve, and reinstall that
version if the resolve moved it.
--no-depskeeps the rest of the resolved setuntouched, and reading the version from the image rather than hardcoding it keeps the
Dockerfile the single place it is chosen.
Why not declare the pin in requirements.txt
Tried it. pip resolves the conflict against torch's exact pin by downgrading torch
itself:
torch 2.11.0+cu130 → 2.10.0 with the whole CUDA stack swapped to cu12. Strictly worse
than the bug.
Measurements
On an H200 devbox running the exact CI image digest
(
radixark/miles@sha256:4d94f611…):pip install -r requirements.txtTE fused attention backward, 8 configs (d=64/128 × MHA/GQA × sbhd/bshd, bf16, causal,
NVTE_FUSED_ATTN=1): 8/8 fail at 9.19.0.56, 8/8 pass at 9.22.0.52.The same devbox runs the full
test_lora_qwen2.5_0.5Bto completion withSelected backend = FusedAttention (sub-backend 1)— the exact path and sub-backendthat fails in CI — once cuDNN is 9.22.0.52.
Related
Depends on #1824, which removes the base image's apt cuDNN. Both are
required: the apt copy shadowed the pip one at load time (mixed sub-libraries,
undefined symbol _ZTVN5cudnn7backend12OperationSetE), and this PR keeps the pip copyat the version the image chose.