docker: stop downgrading cuDNN below what torch ships - #1808
Merged
Conversation
TE 2.17's fused attention calls cudnn-frontend's
Reshape_attributes::set_reshape_mode, which sets the backend attribute
CUDNN_ATTR_OPERATION_RESHAPE_MODE. That attribute exists from cuDNN 9.22.0.
Against the pinned 9.16.0.29 it comes back BAD_PARAM and TE raises out of
fused_attn_bwd:
fused_attn_f16_arbitrary_seqlen.cu:934 cuDNN Error:
set_attribute(..., CUDNN_ATTR_OPERATION_RESHAPE_MODE, ...)
... CUDNN_STATUS_BAD_PARAM
It killed test_qwen3_5_35b_a3b_lora_ci (h100) and test_lora_qwen2.5_0.5B
(h200) in #1795, and reaches only configs where TE does not select FA3, so
most tests never touch it.
The old pin is obsolete in both of its purposes. It existed for
pytorch/pytorch#168167 (Conv3D bf16 collapse on H200), whose maintainers say
the fix landed in PyTorch 2.10 and whose stated workaround for 2.9.x was to
install a *newer* cuDNN, 9.15+. This image runs torch 2.11.0, and torch 2.11
already ships 9.19.0.56, so the pin now only drags cuDNN backwards.
Dropping the pin is not enough: 9.19.0.56 is still below 9.22.0. 9.22.0.52 is
the lowest release that satisfies TE, chosen over the newer 9.23-9.25 to keep
the change minimal.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
yueming-yuan
added
run-ci-image
bypass-fastfail
#1796 switched TransformerEngine to the rolling wheels installed with --no-deps. That pins the three TE dists to exactly those wheels, which is the point, but it also drops transformer_engine_torch's own runtime requirements: einops, onnx, onnxscript, pydantic, nvdlfw-inspect. transformer_engine.pytorch imports onnxscript unconditionally on its core import path (module/__init__ -> layernorm_linear -> base -> _common -> export -> onnx_extensions), so every image built since #1796 fails at ModuleNotFoundError: No module named 'onnxscript' as soon as anything touches TE -- which is every GPU test. verify_transformer_engine.py did not catch it because it only compared versions and metadata and called find_spec('transformer_engine'), which resolves the package directory without importing it. Check that every non-TE requirement of transformer_engine_torch is actually installed, so the build fails at the layer that introduced the gap instead of shipping a green image.
yueming-yuan
changed the title
This was referenced Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is still true: the pin is an obsolete downgrade
docker/Dockerfilepinsnvidia-cudnn-cu13==9.16.0.29under a bare link topytorch/pytorch#168167 (Conv3D bf16
~40,000x slowdown on H200). Both of that pin's reasons have expired:
nWEIdia: "wait for 2.10 for the fix" — this image runs torch 2.11.0eqy: "manually install a newer version of cuDNN (9.15+)" — torch 2.11 already ships 9.19.0.56The pin was originally an upgrade (the base image sat below 9.15). Now that torch ships
9.19.0.56 it has inverted into a downgrade, visible in every build log:
What this does not fix
I claimed 9.22.0.52 resolves the
fused_attn_bwdfailure. It does not. With9.22.0.52 installed (confirmed in the build log of #1795 run
30237617317), the identical error still fires:
4 occurrences in
test_qwen3_5_35b_a3b_lora_ci(h100), 6 intest_lora_qwen2.5_0.5B(h200).
Why this PR's own CI looked clean. It branches off main, which does not carry
radixark/Megatron-Bridge#24. Both affected LoRA tests therefore died earlier, at
ValueError: not enough values to unpack (expected 21, got 16), and never reached theattention backward at all. Zero cuDNN errors here meant the path was never executed,
not that it was fixed. Only #1795 — which pins the merged Megatron-Bridge — gets far
enough to exercise it.
Open
The real requirement is unknown. Candidates still to test: a higher cuDNN (9.23 / 9.24 /
9.25 are on PyPI); or that this is not a version floor at all —
transformer_engine_cu13ships as a prebuilt binary and may not be loading the pip-installed cuDNN it appears to.
Needs a minimal repro on a devbox and a bisect, not another inference from release notes.
Note that TE declares no cuDNN dependency whatsoever (
transformer-engine-cu13requiresonly pydantic / importlib-metadata / packaging), so whatever the constraint is, no
metadata check can enforce it.