Skip to content

docker: keep the cu12 dependency markers after checking out sglang-miles - #1961

Merged
yueming-yuan merged 1 commit into
mainfrom
fix-cu12-pyproject-after-checkout
Jul 30, 2026
Merged

docker: keep the cu12 dependency markers after checking out sglang-miles#1961
yueming-yuan merged 1 commit into
mainfrom
fix-cu12-pyproject-after-checkout

Conversation

@yueming-yuan

Copy link
Copy Markdown
Collaborator

What breaks

Docker Build & Push on main fails at Install sglang-miles:

#38 [sglang 33/39] RUN cd /sgl-workspace/sglang && git fetch origin sglang-miles && ...
 * branch            sglang-miles -> FETCH_HEAD
error: Your local changes to the following files would be overwritten by checkout:
	python/pyproject.toml
Please commit your changes or stash them before you switch branches.

First failure: run 30491453615 (the first release build after #1795).

Why

sglang's own Dockerfile rewrites the dependency markers in place for CUDA 12 images and never commits the result:

# sgl-project/sglang, docker/Dockerfile @ v0.5.16, L713
cd /sgl-workspace/sglang \
&& if [ "${CUDA_VERSION%%.*}" = "12" ]; then \
       sed -i 's/cuda-python>=13\.0/cuda-python>=12,<13/' python/pyproject.toml && \
       sed -i 's/flashinfer_python\[cu13\]/flashinfer_python[cu12]/' python/pyproject.toml && \
       sed -i 's/nvidia-cutlass-dsl\[cu13\]/nvidia-cutlass-dsl/' python/pyproject.toml; \
   fi \
&& python3 -m pip install --no-deps -e "python[${BUILD_TYPE}]"

So v0.5.16-cu129 ships a dirty working tree and our checkout aborts. The release build is the only path that hits this: it pins SGLANG_IMAGE_TAG=v0.5.16-cu129 in docker/build.py, while PR CI uses the default v0.5.16 (cu130), which builds with CUDA_VERSION=13.0.1 and never runs that rewrite. v0.5.15-cu129 was clean, so this is new in the v0.5.16 CUDA 12 image rather than a regression in our tree.

The fix

Force the checkout, then re-apply the same three rewrites when ENABLE_CUDA_13 != 1.

Forcing on its own is not enough. The tree we check out carries the cu13 markers, pip install -e publishes them as sglang's installed metadata, and every later pip resolve then sees unsatisfied cu13 requirements on a cu12 image — the same mechanism that silently downgraded cuDNN in #1836.

Checks

  • The three sed patterns match sglang-miles's python/pyproject.toml (lines 26, 34, 48) and rewrite exactly those three lines.
  • humming-kernels[cu13] is left alone, matching upstream — sglang's own build does not rewrite it either.
  • sglang-miles differs from the v0.5.16 tag in this file only by adding xxhash, so after the rewrite the tree matches the base image's dependency markers plus that one addition.

The CUDA 12 sglang images rewrite cuda-python, flashinfer_python and
nvidia-cutlass-dsl in python/pyproject.toml from cu13 to cu12 at image build time
and never commit it, so `git checkout FETCH_HEAD` aborts on the dirty file and the
release build fails at 'Install sglang-miles'. The cu130 image never runs that
rewrite, which is why only the cu129 release path breaks.

Force the checkout and re-apply the same three rewrites when ENABLE_CUDA_13 != 1.
Forcing alone would leave cu13 markers on a cu12 image, and the editable install
publishes them to every later pip resolve.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@yueming-yuan

Copy link
Copy Markdown
Collaborator Author

Verified against the real lmsysorg/sglang:v0.5.16-cu129 image on a CPU build box, not just read off upstream's Dockerfile.

The dirty file is exactly the three markers

$ git -C /sgl-workspace/sglang status --porcelain
 M python/pyproject.toml

$ git diff -- python/pyproject.toml
-  "cuda-python>=13.0",
+  "cuda-python>=12,<13",
-  "flashinfer_python[cu13]==0.6.14",
+  "flashinfer_python[cu12]==0.6.14",
-  "nvidia-cutlass-dsl[cu13]==4.6.0",
+  "nvidia-cutlass-dsl==4.6.0",

$ git log --oneline -1
fdebc93 [Cherry-pick to release/v0.5.16] Fix stale flashinfer-MLA fallback poisoning spec verify capture (#32288) (#32346)

The failure reproduces verbatim

$ git fetch origin sglang-miles && git checkout FETCH_HEAD
error: Your local changes to the following files would be overwritten by checkout:
	python/pyproject.toml
Please commit your changes or stash them before you switch branches.
Aborting

This PR's sequence resolves it

Running the new block with ENABLE_CUDA_13=0 inside that image:

checkout+sed OK   HEAD=3003d70 [sglang-miles] ...

26:  "cuda-python>=12,<13",
34:  "flashinfer_python[cu12]==0.6.14",
37:  "humming-kernels[cu13]==0.1.10",     <- left alone, as upstream does
48:  "nvidia-cutlass-dsl==4.6.0",
90:  "xxhash",  # /pull_weights delta checksum   <- sglang-miles' own addition, preserved

The tree ends up at sglang-miles with the cu12 markers restored, so the cu13 specifiers never reach sglang's installed metadata on a cu12 image.

Note this PR's own CI exercises the cu130 path only; the cu129 path is covered by the run above and by the release build after merge.

@guapisolo guapisolo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved to unblock.

@yueming-yuan
yueming-yuan merged commit fa25556 into main Jul 30, 2026
10 checks passed
@yueming-yuan
yueming-yuan deleted the fix-cu12-pyproject-after-checkout branch July 30, 2026 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants