Skip to content

[DNM] test SGLang + LMCache server through srt-slurm - #3554

Closed
cquil11 wants to merge 3 commits into
mainfrom
test/dnm-sglang-lmcache-server
Closed

cquil11 wants to merge 3 commits into
mainfrom
test/dnm-sglang-lmcache-server

Conversation

@cquil11

@cquil11 cquil11 commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Test PR, do not merge.

Runs one AgentX point through srt-slurm with SGLang's LMCache unified radix cache (enable-lmcache) against the lmcache-server service from SemiAnalysisAI/srt-slurm#32, applied as 507-lmcache-server-atom-sglang.patch.

  • Qwen3.5-397B-A17B FP8, B200 TP8, conc 76, KV offload to DRAM (the HiCache c76 point with LMCache in its place).
  • lmsysorg/sglang:nightly-dev-cu13-20260928-81f27fb3, LMCache 0.5.6.dev112 (nightly; the first build with the SGLang unified connector).
  • No NEXTN: LMCacheUnifiedRadixCache does not support speculative decoding yet.

Result

Path Hardware Run Result
lmcache-server + SGLang enable-lmcache B200 TP8, conc 76 36490342974 Passed: 1030/1030 requests, cache hit 87% (39% GPU, the rest from LMCache)

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps only run on labeled PRs. Add full-sweep-fail-fast (strongly recommended); use full-sweep-enabled only when matrix jobs should continue after a failure.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**扫描仅在带有标签的 PR 上运行。强烈建议添加 full-sweep-fail-fast;仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@cquil11

cquil11 commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: test-only PR. SGLang + LMCache server path validated on B200 (run 36490342974); result recorded in NVIDIA/srt-slurm#528.

@cquil11 cquil11 closed this Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants