Skip to content

Commit f0e8589

Browse files
Drop CUDA 12.6 support (#24740)
## Summary PyTorch no longer publishes cu126 nightlies. The torchao cu126 channel stops at `0.19.0.dev20260907`, so the 12.6 CI rows can't install the current pins. This removes CUDA 12.6 support. **Support** - `install_utils.py`: remove `(12, 6)` from `SUPPORTED_CUDA_VERSIONS`. Supported versions are now 13.0, 13.2, 13.4. **CI** - `.github/workflows/vulkan.yml`: move `test-vulkan-nvidia` and the Vulkan LLM e2e job from the `cuda12.6` image to `cuda13.2`. The `pytorch/almalinux-builder:cuda13.2` image exists and is current. - `.github/workflows/cuda.yml`: drop `12.6` from the `test-cuda-builds` matrix. Coverage that remains: - build matrix rows for 13.0 and 13.4 - 13.0 on every model e2e / benchmark job (10+ jobs) - 13.2 on the Windows e2e jobs (`cuda-windows.yml`) and the published wheel rows (`filter_cuda_matrix.py`: cu132 / cu134) - `.ci/scripts/wheel/cuda_arch_list.sh`: remove the cu126 GPU arch lists. A cu126 row now fails as an unknown train. - `.ci/docker/common/install_cuda_windows_cross_compile.sh`: remove the 12.6 entry. - `.ci/scripts/tests/test_cuda_workflow.py`, `.ci/scripts/tests/test_cu134_dependencies.py`: drop the 12.6 cases. **Docs / comments** - `docs/source/backends/cuda/cuda-overview.md`: the source build now lists CUDA 13.0 instead of 12.6. - Update the 12.6 / cu126 examples in `install_utils.py`, `setup.py`, `filter_cuda_matrix.py`, `test_shared_libraries.py`, `cuda_benchmark.py`. **Intentionally unchanged** - `backends/cuda/cuda_backend.py` `_setup_cuda_environment_for_fatbin`: the CUDA 12 ptxas branch handles any user torch build on CUDA 12.x below 12.9, not just a 12.6 support row. - `setup.py` / `test_shared_libraries.py` CUDA 12 runtime package mapping (`nvidia-cuda-runtime-cu12`): this is keyed by CUDA major, not 12.6. - Windows cross-compile entries for 12.8 / 12.9. ## Test plan - `pytest .ci/scripts/tests/test_cu134_dependencies.py .ci/scripts/tests/test_cuda_workflow.py .ci/scripts/tests/test_filter_cuda_matrix.py`: 49 passed - `executorch_cuda_arch_list`: cu130/cu132/cu134 still resolve, cu126 now returns 64 (unknown train) - Both workflow YAMLs parse; flake8 clean - Repo-wide grep for `12.6` / `cu126` / `(12, 6)`: two hits left, both kept on purpose: the `nvidia-cuda-runtime-cu12==12.6.77` example in a parsing comment, and the `CUDA 12.6+` comment on the unchanged ptxas branch in `cuda_backend.py` - CI (Vulkan real-GPU jobs on the 13.2 image) --------- Co-authored-by: Digant Desai <digantdesai@meta.com>
1 parent 3f3a47e commit f0e8589

12 files changed

Lines changed: 25 additions & 63 deletions

File tree

‎.ci/docker/common/install_cuda_windows_cross_compile.sh‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,6 @@ INSTALL_DIR="${WINDOWS_CUDA_INSTALL_DIR:-/opt/cuda-windows}"
1414
# Mapping of CUDA versions to their corresponding driver versions for Windows installers
1515
# Source: https://developer.nvidia.com/cuda-toolkit-archive
1616
declare -A CUDA_DRIVER_MAP=(
17-
["12.6"]="12.6.3:561.17"
1817
["12.8"]="12.8.1:572.61"
1918
["12.9"]="12.9.1:576.57"
2019
["13.0"]="13.0.2:"

‎.ci/scripts/cuda_benchmark.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -588,7 +588,7 @@ def to_v3_format(
588588
workflow_run_id: GitHub workflow run ID
589589
workflow_run_url: GitHub workflow run URL
590590
gpu_name: GPU device name (e.g., "Tesla V100", "A100")
591-
cuda_driver_version: CUDA driver version (e.g., "12.6", "535.104.05")
591+
cuda_driver_version: CUDA driver version (e.g., "13.0", "535.104.05")
592592
593593
Returns:
594594
List of benchmark records in v3 format
@@ -873,7 +873,7 @@ def main():
873873
"--cuda_driver_version",
874874
type=str,
875875
default=None,
876-
help="CUDA driver version (e.g., '12.6', '535.104.05') - optional for v3 format",
876+
help="CUDA driver version (e.g., '13.0', '535.104.05') - optional for v3 format",
877877
)
878878
parser.add_argument("--verbose", action="store_true", help="Print verbose output")
879879

‎.ci/scripts/tests/test_cu134_dependencies.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ def test_all_install_steps_preserve_exact_cu134_selection(self):
103103
)
104104

105105
def test_other_cuda_trains_keep_existing_pins(self):
106-
for cuda in ((12, 6), (13, 0), (13, 2)):
106+
for cuda in ((13, 0), (13, 2)):
107107
for machine in ("x86_64", "aarch64"):
108108
with self.subTest(cuda=cuda, machine=machine):
109109
core, local, domains, examples = self.install_commands(
@@ -159,7 +159,7 @@ def test_cpu_release_dependencies_are_opt_in(self):
159159
)
160160

161161
def test_cpu_release_option_keeps_cuda_pins(self):
162-
for cuda in ((12, 6), (13, 0), (13, 2), (13, 4)):
162+
for cuda in ((13, 0), (13, 2), (13, 4)):
163163
for machine in ("x86_64", "aarch64"):
164164
with self.subTest(cuda=cuda, machine=machine):
165165
self.assertEqual(

‎.ci/scripts/tests/test_cuda_workflow.py‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,7 @@ def _model_quant(entry):
3434
class CudaWorkflowTest(unittest.TestCase):
3535
def test_build_matrix_preserves_existing_cuda_versions(self):
3636
job = WORKFLOW["jobs"]["test-cuda-builds"]
37-
self.assertEqual(
38-
job["strategy"]["matrix"]["cuda-version"], ["12.6", "13.0", "13.4"]
39-
)
37+
self.assertEqual(job["strategy"]["matrix"]["cuda-version"], ["13.0", "13.4"])
4038
self.assertEqual(job["with"]["gpu-arch-version"], "${{ matrix.cuda-version }}")
4139

4240
def test_cuda_builds_take_the_node_driver_on_an_unpinned_v3(self):
@@ -58,9 +56,7 @@ def test_cuda_runtime_update_precedes_build_and_propagates_failure(self):
5856
source() { printf 'BUILD %s\n' "$*"; printf 'LIBRARY PATH %s\n' "$LD_LIBRARY_PATH"; }
5957
"""
6058
for version, conda_status in (
61-
(version, status)
62-
for version in ("12.6", "13.0", "13.4")
63-
for status in (0, 1)
59+
(version, status) for version in ("13.0", "13.4") for status in (0, 1)
6460
):
6561
with self.subTest(version=version, conda_status=conda_status):
6662
result = subprocess.run(

‎.ci/scripts/wheel/cuda_arch_list.sh‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -41,29 +41,6 @@ _cuda_arch_aarch64_cu130="8.0 9.0 10.0 11.0 12.0"
4141
_cuda_arch_aarch64_cu132="${_cuda_arch_aarch64_cu130}"
4242
_cuda_arch_aarch64_cu134="${_cuda_arch_aarch64_cu130}"
4343

44-
# The older CUDA train.
45-
#
46-
# The two architectures do not carry identical lists, because each covers what the published PyTorch
47-
# build for that architecture covers, and those differ. Matching them to each other instead would mean
48-
# advertising a GPU on one architecture that PyTorch cannot serve there.
49-
#
50-
# The smaller embedded modules are deliberately absent, with one exception. An embedded-only
51-
# architecture in a generic wheel would advertise a device the row cannot otherwise serve, since
52-
# those devices also need the CUDA, TensorRT and PyTorch pinned by their own software release
53-
# rather than the ones a generic wheel resolves.
54-
#
55-
# 8.7 is that exception. This is the only row whose CUDA major matches what that module's software
56-
# release ships, and the wheel declares no PyTorch, so the user supplies the build that carries
57-
# their architecture. Omitting it does not protect them from a bad pairing, it only removes the
58-
# device code they need.
59-
#
60-
# The floor is 8.0 rather than the oldest architecture PyTorch still carries. One of these sources compiles
61-
# an integer matrix-multiply path only at 8.0 and newer, so an older architecture would get a delegate that
62-
# loads, runs most models, and fails on one that needs that operator. Claiming hardware the delegate only
63-
# partly serves is the same problem the embedded modules have, so the row leaves it out for the same reason.
64-
_cuda_arch_x86_64_cu126="8.0 8.6 8.9 9.0"
65-
_cuda_arch_aarch64_cu126="8.0 8.7 9.0"
66-
6744
# A CUDA train with no architecture list would leave the build detecting the builder's GPU, which is
6845
# the failure this file exists to prevent. Adding a train to the release matrix without adding its
6946
# architectures should fail loudly rather than silently produce a single-GPU wheel.
@@ -103,7 +80,6 @@ executorch_cuda_arch_list() {
10380
case "${machine}" in
10481
aarch64 | arm64)
10582
case "${train}" in
106-
126) printf '%s' "${_cuda_arch_aarch64_cu126}" ;;
10783
130) printf '%s' "${_cuda_arch_aarch64_cu130}" ;;
10884
132) printf '%s' "${_cuda_arch_aarch64_cu132}" ;;
10985
134) printf '%s' "${_cuda_arch_aarch64_cu134}" ;;
@@ -112,7 +88,6 @@ executorch_cuda_arch_list() {
11288
;;
11389
x86_64)
11490
case "${train}" in
115-
126) printf '%s' "${_cuda_arch_x86_64_cu126}" ;;
11691
130) printf '%s' "${_cuda_arch_x86_64_cu130}" ;;
11792
132) printf '%s' "${_cuda_arch_x86_64_cu132}" ;;
11893
134) printf '%s' "${_cuda_arch_x86_64_cu134}" ;;

‎.ci/scripts/wheel/test_shared_libraries.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1217,7 +1217,7 @@ def _wheel_cuda_train() -> str:
12171217
"""The CUDA train the installed wheel was built for, or "" for a CPU wheel.
12181218
12191219
Read from the local version segment, which is the only place the wheel states what
1220-
it was built for. `1.5.0+cu126` gives "126".
1220+
it was built for. `1.5.0+cu132` gives "132".
12211221
"""
12221222
local = importlib.metadata.version("executorch").partition("+")[2]
12231223
return local[2:] if local.startswith("cu") else ""
@@ -1360,7 +1360,7 @@ def _resolve_required(required):
13601360
),
13611361
# The CUDA components. Required exactly when the wheel says it is a CUDA wheel,
13621362
# which is decided at check time rather than here: a fixed False meant a wheel
1363-
# tagged +cu126 carrying no CUDA library at all passed every check in this file,
1363+
# tagged +cu132 carrying no CUDA library at all passed every check in this file,
13641364
# while a fixed True would fail every CPU wheel. The marker is the string these
13651365
# rows are keyed on below.
13661366
(
@@ -1457,7 +1457,7 @@ def test_each_component_has_one_owner() -> None:
14571457
shipped = {path.name for path in _shipped_shared_objects(_installed_package_dir())}
14581458
for what, symbols, owner, required in _OWNED_COMPONENTS:
14591459
# Resolved here rather than in the table, because it depends on the installed
1460-
# wheel. A fixed False let a wheel tagged +cu126 ship with no CUDA library at
1460+
# wheel. A fixed False let a wheel tagged +cu132 ship with no CUDA library at
14611461
# all and still pass, which is the whole point of the conditional rows.
14621462
required = _resolve_required(required)
14631463
present = any(name.startswith(owner) for name in shipped)

‎.github/scripts/filter_cuda_matrix.py‎

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -53,12 +53,6 @@
5353
# Skip wholly absent trains so an upstream removal cannot block the remaining releases.
5454
# Offered trains must still cover every supported Python version.
5555
#
56-
# cu126 was the floor until PyTorch stopped offering it on the nightly channel. The generator
57-
# still offers it when a release is cut, so keeping it here would publish a train that no
58-
# nightly has built since, and a release is the wrong place to find out that it no longer
59-
# builds. A machine on CUDA 12.6 can still build from source, where the pinned torch comes
60-
# from a channel that carries 12.6.
61-
#
6256
# cu130 is not published. PyTorch keeps it on its nightly builds only as a temporary hold for
6357
# other projects, and its release-candidate builds do not carry it, so a release would have
6458
# no cu130 train even while nightlies do.
@@ -91,9 +85,8 @@
9185
# version. Kept empty on purpose today, so no Jetson row is emitted.
9286
#
9387
# Published PyTorch stopped shipping sm_87 device code after 2.8.0, so a Jetson row today
94-
# would produce a wheel whose PyTorch dependency cannot execute on the device. The aarch64
95-
# CUDA 12.6 row, which was the one that compiled sm_87 device code for an embedded module,
96-
# is no longer published either. Populate this when both change.
88+
# would produce a wheel whose PyTorch dependency cannot execute on the device, and no
89+
# published row compiles sm_87 device code. Populate this when both change.
9790
#
9891
# Because both lists are empty, asking for the JetPack rows can only produce an empty result.
9992
# No workflow asks, and the request is rejected up front with that reason rather than left to

‎.github/workflows/cuda.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Test ExecuTorch CUDA Build Compatibility
22
# This workflow tests whether ExecuTorch can be successfully built with CUDA support
3-
# across different CUDA versions (12.6, 13.0, 13.4) using the command:
3+
# across different CUDA versions (13.0, 13.4) using the command:
44
# ./install_executorch.sh
55
#
66
# It also runs model accuracy (e2e EXPECTED_OUTPUT checks) and, for the perf
@@ -142,7 +142,7 @@ jobs:
142142
strategy:
143143
fail-fast: false
144144
matrix:
145-
cuda-version: ["12.6", "13.0", "13.4"]
145+
cuda-version: ["13.0", "13.4"]
146146

147147
name: test-executorch-cuda-build-${{ matrix.cuda-version }}
148148
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
@@ -159,8 +159,8 @@ jobs:
159159
script: |
160160
set -eux
161161
162-
# AOTI model libraries need a newer C++ runtime than the 12.6 image's
163-
# system library; the 13.4 image also needs the thread-local cleanup fix.
162+
# AOTI model libraries need a newer C++ runtime than the image's system
163+
# library; the 13.4 image also needs the thread-local cleanup fix.
164164
conda install -y -n base -c conda-forge 'libstdcxx-ng=16.2.0' 'libgcc-ng=16.2.0'
165165
export LD_LIBRARY_PATH="/opt/conda/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
166166
source .ci/scripts/test-cuda-build.sh "${{ matrix.cuda-version }}"

‎.github/workflows/vulkan.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ jobs:
7979
timeout: 120
8080
runner: mt-l-x86aavx2-11-41-a10g
8181
gpu-arch-type: cuda
82-
gpu-arch-version: "12.6"
82+
gpu-arch-version: "13.2"
8383
submodules: recursive
8484
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
8585
script: |
@@ -171,7 +171,7 @@ jobs:
171171
timeout: 180
172172
runner: mt-l-x86aavx2-11-41-a10g
173173
gpu-arch-type: cuda
174-
gpu-arch-version: "12.6"
174+
gpu-arch-version: "13.2"
175175
submodules: recursive
176176
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
177177
upload-artifact: vulkan-llm-e2e

‎docs/source/backends/cuda/cuda-overview.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Below are the requirements for running a CUDA-delegated ExecuTorch model:
1717
- **Hardware**: NVIDIA GPU with compute capability 8.0 (Ampere) or newer. This is the floor the
1818
prebuilt packages are built for.
1919
- **CUDA**: 13 for the prebuilt packages. On Linux they bring the CUDA runtime with them. On
20-
Windows it comes from a CUDA 13 toolkit install. A build from source also supports CUDA 12.6.
20+
Windows it comes from a CUDA 13 toolkit install. A build from source also supports CUDA 13.0.
2121
- **Operating System**: Linux, or Windows from C++ for programs exported on Linux (see
2222
[CUDA on Windows](../../using-executorch-cpp.md#cuda-on-windows))
2323
- **Drivers**: An NVIDIA driver that supports the CUDA version you use: CUDA 13 for the prebuilt

0 commit comments

Comments
 (0)