Repository navigation
feat: support the LMCache server on ATOM and SGLang - #528
Merged
ishandhanani merged 7 commits intoSep 29, 2026
Merged
Conversation
cquil11
requested review from
alec-flowers,
csahithi,
hjjq,
ishandhanani,
kedarpotdar-nv,
kyleliang-nv,
nlevin-ui and
qiching
as code owners
September 29, 2026 05:01
A role can list extra-kv-connectors in its args. srtctl keeps generating the Mooncake P/D connector with its handshake port and wraps it with the extras in ATOM's multi connector, which is how ATOM runs its in-process LMCache CPU offload on prefill. A single extra connector on an aggregate role is passed unwrapped. Signed-off-by: Cameron Quilici <cjquilici@gmail.com>
…server SGLang's enable-lmcache flag runs LMCacheUnifiedRadixCache over LMCache's multiprocess connector, which reads the server address from LMCACHE_MP_HOST and LMCACHE_MP_PORT. srtctl now sets both to the lmcache-server service on the worker's own node, unless the recipe passes lmcache-config-file or sets LMCACHE_MP_HOST in the role env. Signed-off-by: Cameron Quilici <cjquilici@gmail.com>
Drop the aggregate-role path, the underscore spelling and per-entry validation; prefill/decode wrap Mooncake plus the extras in multi and aggregate roles reject extra-kv-connectors. Signed-off-by: Cameron Quilici <cjquilici@gmail.com>
ATOM runs lmcache_offload on aggregate servers too, and its lmcache_mp connector is a client for the same lmcache server that the lmcache-server service launches. extra-kv-connectors now work on any role: an aggregate role with one connector runs it alone, and several are wrapped in multi. lmcache_mp defaults lmcache.mp.port to srtctl's node-local server port unless the recipe gives an address. Signed-off-by: Cameron Quilici <cjquilici@gmail.com>
Signed-off-by: Cameron Quilici <cjquilici@gmail.com>
Signed-off-by: Cameron Quilici <cjquilici@gmail.com>
Signed-off-by: Cameron Quilici <cjquilici@gmail.com>
cquil11
force-pushed
the
feat/lmcache-atom-sglang
branch
from
September 29, 2026 16:16
9b65d1a to
71936ee
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #528 +/- ##
=======================================
Coverage ? 83.39%
=======================================
Files ? 154
Lines ? 22141
Branches ? 0
=======================================
Hits ? 18465
Misses ? 3676
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the LMCache service from #507 to ATOM and SGLang, and fixes connector forwarding for direct vLLM aggregate workers. Includes #507's commit as the base.
extra-kv-connectors, preserving Mooncake for P/D transfer. Defaultslmcache_mpto the service's port, 8750; explicit ports take precedence.enable-lmcacheis enabled, unless a config file or host override is provided.--kv-transfer-configwhen launching withvllm serve.Includes example recipes, launch snapshots, and regression tests.
Validation
lmcache_offloadlmcache-mp+lmcache-serverenable-lmcache+lmcache-server