Repository navigation
Add Qwen3.5 FP8 GB300 AgentX DEP16 recipes / 新增 Qwen3.5 FP8 GB300 AgentX DEP16 配方 - #3840
Conversation
新增 Qwen3.5 FP8 GB300 AgentX DEP16 配方及所选配置。
将 Qwen3.5 DEP16 变更记录关联到 PR。
|
Thanks for the contribution!
中文感谢你的贡献！
|
RohitNagraj
added
the
full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any confirmed bugs. Because it touches CODEOWNER-protected paths (inferencex-e2e/configs/nvidia-master.yaml and the gb300-* srt-slurm recipe tree) with new speculative-decoding (MTP6) and HiCache wiring, a human look from one of the listed owners would still be worthwhile.
What was reviewed: the model.container/image and Dynamo-revision parity between the new recipe and master config (both match); the new agg-midcurve search-space points against the pre-existing agg-pareto sweep (three of them reuse the same CONFIG_FILE/conc-list, which is redundant but not incorrect); the benchmark.use_chat_template: true field against the agentic-replay chat-template env var (AIPERF_APPLY_CHAT_TEMPLATE) it doesn't actually wire up — but no sibling multi-node AgentX MTP recipe sets that env var either, so this isn't a regression this PR introduces; and the perf-changelog.yaml entry (single tail-appended block, English, covering both new config keys).
Extended reasoning...
Pure-addition config/recipe change (449 lines, 3 files): a new GB300 FP8 Qwen3.5 AgentX disaggregated recipe with MTP6 speculative decoding and HiCache offload, its wiring into two new nvidia-master.yaml search-space keys, and one perf-changelog.yaml append. No auth/crypto/data-exposure surface; the main risk is benchmark-config correctness (precision/env settings, search-space duplication) rather than security. Decisive factors for defer over approve: both changed files fall under CODEOWNERS entries (nvidia-master.yaml and gb300-* recipes), and independent checks surfaced a chat-template env-var mismatch and duplicate sweep points that, while judged non-blocking (pre-existing pattern / minor redundancy, not a regression), are worth an owner's confirmation.
This review covers commit 94e50cd, which is no longer the latest commit on this pull request; later commits are not covered by it.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=37833971361 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=37833971361 |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs ( |
|
[by Claude Code] sgl-project/sglang#43083 has been merged. This PR's AgentX configs ( This PR has no authorized |
|
/use 37833971361 |
|
@nvpohanh staged run 37833971361: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-10-08~r37833971361 This run remains available across future |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Passing sweep on in-PR commit): PASS — run 37833971361 ran on head ✅ Check 2 (Evals): PASS — in ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#43083 is ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist): PASS — every item and sub-item of the current ✅ Check 6 (Upstream images / engine ordering): PASS — both new entries are ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (Architecture hacks / event publication): PASS — no ✅ Check 9 (Chat template): PASS — ✅ Check 10 (Engine patches): PASS — no patch files, ✅ Check 11 (Agentic golden AL): PASS — job logs show ➖ Check 12 (Append-only): N/A — the new changelog entry does not set ✅ Check 13 (Draft precision): PASS — the draft is the embedded MTP head of ✅ Check 14 (Pareto coverage): PASS — one curve, ✅ Check 15 (PR description): PASS — the body's claims (1/2/3 PP4 prefill with a DEP16 decode, MTP6, HiCache, decode splits 1, decode Assessed commit: |
cquil11
left a comment
cquil11
left a comment
There was a problem hiding this comment.
Approved for merge with the existing CODEOWNER sign-off and sweep reuse. Audited all eight throughput points in https://github.com/SemiAnalysisAI/InferenceX/actions/runs/37833971361: the actual AIPerf profiles use the 256k dataset, and profiling server-log AL interval means closely match the synthetic targets (4.41 or 4.80).
…s from the pareto bundles #3840's midcurve entries rerun two disaggregated and three aggregated points of the pareto entries by naming those points' standalone recipes, which the bundles replace. Their rows select the matching pareto overrides, so each still runs exactly the pareto point's recipe rather than a copy that could drift; the DEP16 points keep disagg-dep16-mtp6-variants.yaml. RECIPES.md notes that an entry rerunning another entry's points selects its overrides.
3 participants
Add three Qwen3.5-397B-A17B-FP8 GB300 AgentX configurations pairing one, two, or three PP4 prefill workers with one TP/DP/EP16 decode worker. The shared recipe bundle uses MTP6, HiCache, decode sequence-length splits 1, and
max-mamba-cache-size: 1872for decode.Two matrix entries select five disaggregated configurations and three existing aggregate configurations on
lmsysorg/sglang:nightly-dev-cu13-20260913-fa663e72, retaining Dynamo revisionad6f7d54ce7b99c53576b10482755c93c921fbac. PP4 prefill roles setPYTHONOPTIMIZE=1, which disables Python assertions, to allow MTP with overlap scheduling on this nightly; decode and aggregate roles retain normal assertion behavior.中文
新增三种 Qwen3.5-397B-A17B-FP8 GB300 AgentX 配置，分别将一、二、三个 PP4 prefill worker 与一个 TP/DP/EP16 decode worker 配对。共享配方使用 MTP6、HiCache、decode sequence-length splits 1，并将 decode 的
max-mamba-cache-size设为 1872。两个 matrix 条目选择五种分离式配置和三种已有聚合配置，使用
lmsysorg/sglang:nightly-dev-cu13-20260913-fa663e72，并沿用 Dynamo 固定版本ad6f7d54ce7b99c53576b10482755c93c921fbac。PP4 prefill 角色设置PYTHONOPTIMIZE=1，禁用 Python 断言，以便在此 nightly 中同时使用 MTP 和 overlap scheduling；decode 和聚合角色保留正常的断言行为。AI model disclosure
gpt-6.1-sol: prepared the recipe settings and matrix configuration.claude-opus-5-5: reviewed the recipe settings and selected configurations.AI 模型说明：
gpt-6.1-sol编写配方参数和 matrix 配置；claude-opus-5-5审阅配方参数及所选配置；Codex 整理配方变体、准备配置及变更记录、执行本地检查并起草本说明，当前精确模型及版本无法核实。