ci: CUDA build improvements

[kenvandine]

Three changes, all measured against run 30332616539.

1. Cache the windows-latest-cmake build

Its cuda12 leg compiles nine CUDA architectures (61;70;75;80;86;89;90;100;120) in a single job and had no ccache step at all. It was the second-longest job in the run, behind only ROCm:

  4.8 min  Install cuda-toolkit
 90.6 min  Build          ← uncached

For comparison, windows-latest-cuda runs the same MSVC / Ninja / Release setup with ccache and averages 1.6 min of build per leg at 97.96% hit rate.

No cmake flags are needed. ggml's src/CMakeLists.txt:68 sets RULE_LAUNCH_COMPILE as soon as it finds ccache on PATH (GGML_CCACHE defaults to ON), which is exactly how windows-latest-cuda gets its hits today — it doesn't pass -DCMAKE_*_COMPILER_LAUNCHER either:

if (GGML_CCACHE AND NOT CMAKE_C_COMPILER_LAUNCHER AND NOT CMAKE_CXX_COMPILER_LAUNCHER)
    find_program(GGML_CCACHE_FOUND ccache)
    ...
        set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "${GGML_CCACHE_VARIANT}")

The cuda12 leg gets max-size: 2G; the 500M default that suits cpu and vulkan would thrash, since a single-architecture CUDA cache already measures 0.1 GB and this builds nine plus GGML_CPU_ALL_VARIANTS=ON.

2. Install only the CUDA components the Linux builds use

Both Linux CUDA jobs installed the cuda-toolkit-12-9 metapackage — apt reports 3935 MB of archives per job, on 8 x64 legs plus arm64 every night. That made Install CUDA Toolkit the second-largest cost in those jobs:

ubuntu-latest-cuda (8 legs)	total	avg
Pack artifacts	50.9 min	6.4 min
Install CUDA Toolkit	36.5 min	4.6 min
Build	12.8 min	1.6 min

Most of it is never used. The metapackage pulls Nsight Systems/Compute, cuFFT, cuSPARSE, cuSOLVER and NPP. Checked against source rather than assumed — ggml-cuda/CMakeLists.txt links only:

target_link_libraries(ggml-cuda PRIVATE CUDA::cudart CUDA::cublas)
target_link_libraries(ggml-cuda PRIVATE CUDA::cuda_driver)
target_link_libraries(ggml-cuda PRIVATE CCCL::CCCL)

and stable-diffusion.cpp links only ggml (target_link_libraries(${SD_LIB} PUBLIC ggml)). No reference to cuFFT/cuSPARSE/cuSOLVER/NPP/NVTX in either tree.

So: cuda-nvcc, cuda-cudart-dev (also supplies the libcuda stub behind CUDA::cuda_driver), cuda-cccl for thrust/cub, libcublas-dev, and the libcurand-dev / libnvjitlink-dev libs the Bundle step copies — ~595 MB of direct archives. All six verified present in both repos the jobs use:

=== ubuntu2204/x86_64 ===        === ubuntu2404/sbsa ===
cuda-nvcc-12-9         38.7 MB   34.0 MB
cuda-cudart-dev-12-9    1.4 MB    1.4 MB
cuda-cccl-12-9          1.0 MB    1.0 MB
libcublas-dev-12-9    482.9 MB  483.0 MB
libcurand-dev-12-9     45.6 MB   45.6 MB
libnvjitlink-dev-12-9  25.8 MB   24.8 MB

Added a guard that names the missing file if NVIDIA ever repackages one of these, rather than failing partway through the compile.

The Windows CUDA installs already select sub-packages via Jimver/cuda-toolkit and were left alone.

3. Parallelize xz when packing the Linux CUDA artifacts

tar -cJf drives a single-threaded xz — 50.9 min across the eight x64 legs plus 5.9 min on arm64, four times the compile time. Measured on a 2.9 GB binary payload:

	time	size
xz -T1 (runner default)	12m41s	719.8 MB
xz -T0 (32 cores)	47.7s	732.4 MB

Round-trip verified — xz -t passes, diff -r on the extracted tree is clean. Two caveats: the archive is 1.8% larger, because threaded mode compresses independent blocks; and the runners are 4-core, not 32, so expect roughly 3–4× rather than 16×.

Checked and deliberately not changed

• ccache health. 98–100% hit rates on every job that already has it, every restore succeeding. Nothing to fix.
• Release artifact download. 27 artifacts / 11.71 GiB, but on a full ubuntu-latest VM it completes in 1.4 min. (The equivalent step in ci: unblock the release job's artifact download, parallelize xz, and slim the CUDA install llama.cpp#25 was stalling only because that job ran on ubuntu-slim, a container runner without the disk for it.)
• evict-old-files: 1d. Looks risky against a nightly cron, but isn't. Scheduled runs actually start between 05:42 and 06:30 despite 0 3 * * *, so gaps regularly exceed 24h — 24h23m and 24h24m in the last week — and hit rates on those runs were 99.48% and 100%. ccache --evict-older-than runs in the post step and ccache refreshes mtimes on hits, so the saved snapshot always contains what the run just used.
• Free disk space (8.6 min/run) is now probably unnecessary given the much smaller toolkit, but removing a safety step without measuring the new headroom seemed like the wrong trade. Worth revisiting once this has run a few nights.

Known follow-ups, not in this PR

• The Actions cache is at 10.16 GiB across 93 entries, over the 10 GiB repo limit, so GitHub is already LRU-evicting. Adding the cuda12 cache increases that pressure. It's survivable — the measured 98–100% hit rates happened at the limit, because LRU keeps the nightly-refreshed entries hot — but worth watching. The cheapest headroom is elsewhere: a 1.34 GiB rocm-wheels blob and five retained generations of ccache-ubuntu-rocm-cmake-7.14.0-x64 (317 + 242 + 241 + 209 + 203 MiB ≈ 1.2 GiB of near-duplicates). I did not add a cache-pruning step — deleting caches out from under concurrent jobs is easy to get wrong and there's no evidence yet that it's needed.
• ubuntu-latest-rocm is the real long pole at 167.8 min, and nothing here touches it.
• ubuntu-latest-cmake, ubuntu-latest-cmake-vulkan and the cpu/vulkan legs also lack ccache, but at 5–17 min each they weren't worth the extra cache entries right now. (The cpu and vulkan legs do pick it up here, since the step is on the shared windows-latest-cmake job.)

Testing

Workflow YAML parses; both rewritten Install CUDA Toolkit scripts pass bash -n; the windows-latest-cmake step order is correct (ccache lands before Build, so it's on PATH at configure time); max-size verified as a real input on ggml-org/ccache-action@v1.2.16. All three changes exercise on PR runs, though the first run will be a cold cache — the ccache win shows on the second.

🤖 Generated with Claude Code

[kenvandine]

Summary of Changes

1. ccache on windows-latest-cmake

   • Added ggml-org/ccache-action@v1.2.16 to the windows-latest-cmake job prior to the build step.
   • Sets max-size: 2G for the cuda12 matrix leg (compiling 9 CUDA architectures + CPU variants) and 500M for CPU/Vulkan legs.

2. Minimal CUDA Toolkit APT Dependencies (Linux x64 & ARM64)

   • Replaced full cuda-toolkit-12-9 metapackage (~3.9 GB) with specific required components: cuda-nvcc-12-9, cuda-cudart-dev-12-9, cuda-cccl-12-9, libcublas-dev-12-9, libcurand-dev-12-9, libnvjitlink-dev-12-9 (~1.2 GB).
   • Added an explicit post-install verification loop checking presence of required binaries (nvcc, libcudart, libcublas, libcublasLt, libcurand, libnvJitLink) and headers (cublas_v2.h, cub.cuh).

3. Parallelized xz Compression

   • Replaced tar -cJf with tar -I 'xz -T0' -cf in both ubuntu-latest-cuda and ubuntu-arm64-cuda artifact packaging steps to utilize all available CPU cores.

---

Key Findings & Verification

• Correctness & Safety: The minimal package selection matches all headers and shared libraries required by ggml-cuda and stable-diffusion.cpp CMake target linking and bundling scripts. The added assertion loop ensures early failure with explicit error messages should NVIDIA package structures change in future toolkit releases.
• Performance: Substantially reduces APT archive download/install volume (~2.7 GB saved per Linux CUDA run) and drastically speeds up tar packaging via multi-threaded xz.
• Workflow Syntax: All GitHub Actions expressions, shell constructs, and flag usages (tar -I 'xz -T0') are valid and standard across Ubuntu GitHub runner environments.