docker: keep the cu12 dependency markers after checking out sglang-miles - #1961
Merged
Conversation
The CUDA 12 sglang images rewrite cuda-python, flashinfer_python and nvidia-cutlass-dsl in python/pyproject.toml from cu13 to cu12 at image build time and never commit it, so `git checkout FETCH_HEAD` aborts on the dirty file and the release build fails at 'Install sglang-miles'. The cu130 image never runs that rewrite, which is why only the cu129 release path breaks. Force the checkout and re-apply the same three rewrites when ENABLE_CUDA_13 != 1. Forcing alone would leave cu13 markers on a cu12 image, and the editable install publishes them to every later pip resolve.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Collaborator
Author
|
Verified against the real The dirty file is exactly the three markers The failure reproduces verbatim This PR's sequence resolves it Running the new block with The tree ends up at sglang-miles with the cu12 markers restored, so the cu13 specifiers never reach sglang's installed metadata on a cu12 image. Note this PR's own CI exercises the cu130 path only; the cu129 path is covered by the run above and by the release build after merge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What breaks
Docker Build & Pushon main fails atInstall sglang-miles:First failure: run 30491453615 (the first release build after #1795).
Why
sglang's own Dockerfile rewrites the dependency markers in place for CUDA 12 images and never commits the result:
So
v0.5.16-cu129ships a dirty working tree and our checkout aborts. The release build is the only path that hits this: it pinsSGLANG_IMAGE_TAG=v0.5.16-cu129indocker/build.py, while PR CI uses the defaultv0.5.16(cu130), which builds withCUDA_VERSION=13.0.1and never runs that rewrite.v0.5.15-cu129was clean, so this is new in the v0.5.16 CUDA 12 image rather than a regression in our tree.The fix
Force the checkout, then re-apply the same three rewrites when
ENABLE_CUDA_13 != 1.Forcing on its own is not enough. The tree we check out carries the cu13 markers,
pip install -epublishes them as sglang's installed metadata, and every later pip resolve then sees unsatisfied cu13 requirements on a cu12 image — the same mechanism that silently downgraded cuDNN in #1836.Checks
sglang-miles'spython/pyproject.toml(lines 26, 34, 48) and rewrite exactly those three lines.humming-kernels[cu13]is left alone, matching upstream — sglang's own build does not rewrite it either.sglang-milesdiffers from thev0.5.16tag in this file only by addingxxhash, so after the rewrite the tree matches the base image's dependency markers plus that one addition.