Repository navigation
Add Qwen3.5 FP8 GB300 AgentX DEP16 recipes / 新增 Qwen3.5 FP8 GB300 AgentX DEP16 配方 - #3840
Merged
Merged
Conversation
新增 Qwen3.5 FP8 GB300 AgentX DEP16 配方及所选配置。
RohitNagraj
requested review from
Ankur-singh,
kedarpotdar-nv,
nvpohanh and
wzhao18
as code owners
October 8, 2026 19:40
将 Qwen3.5 DEP16 变更记录关联到 PR。
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
RohitNagraj
added
the
full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
I reviewed this PR and didn't find any confirmed bugs. Because it touches CODEOWNER-protected paths (inferencex-e2e/configs/nvidia-master.yaml and the gb300-* srt-slurm recipe tree) with new speculative-decoding (MTP6) and HiCache wiring, a human look from one of the listed owners would still be worthwhile.
What was reviewed: the model.container/image and Dynamo-revision parity between the new recipe and master config (both match); the new agg-midcurve search-space points against the pre-existing agg-pareto sweep (three of them reuse the same CONFIG_FILE/conc-list, which is redundant but not incorrect); the benchmark.use_chat_template: true field against the agentic-replay chat-template env var (AIPERF_APPLY_CHAT_TEMPLATE) it doesn't actually wire up — but no sibling multi-node AgentX MTP recipe sets that env var either, so this isn't a regression this PR introduces; and the perf-changelog.yaml entry (single tail-appended block, English, covering both new config keys).
Extended reasoning...
Pure-addition config/recipe change (449 lines, 3 files): a new GB300 FP8 Qwen3.5 AgentX disaggregated recipe with MTP6 speculative decoding and HiCache offload, its wiring into two new nvidia-master.yaml search-space keys, and one perf-changelog.yaml append. No auth/crypto/data-exposure surface; the main risk is benchmark-config correctness (precision/env settings, search-space duplication) rather than security. Decisive factors for defer over approve: both changed files fall under CODEOWNERS entries (nvidia-master.yaml and gb300-* recipes), and independent checks surfaced a chat-template env-var mismatch and duplicate sweep points that, while judged non-blocking (pre-existing pattern / minor redundancy, not a regression), are worth an owner's confirmation.
This review covers commit 94e50cd, which is no longer the latest commit on this pull request; later commits are not covered by it.
Contributor
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=37833971361 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=37833971361 |
Collaborator
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs ( |
Collaborator
|
[by Claude Code] sgl-project/sglang#43083 has been merged. This PR's AgentX configs ( This PR has no authorized |
nvpohanh
approved these changes
Oct 9, 2026
Collaborator
|
/use 37833971361 |
Contributor
|
@nvpohanh staged run 37833971361: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-10-08~r37833971361 This run remains available across future |
Collaborator
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
Contributor
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Passing sweep on in-PR commit): PASS — run 37833971361 ran on head ✅ Check 2 (Evals): PASS — in ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#43083 is ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist): PASS — every item and sub-item of the current ✅ Check 6 (Upstream images / engine ordering): PASS — both new entries are ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (Architecture hacks / event publication): PASS — no ✅ Check 9 (Chat template): PASS — ✅ Check 10 (Engine patches): PASS — no patch files, ✅ Check 11 (Agentic golden AL): PASS — job logs show ➖ Check 12 (Append-only): N/A — the new changelog entry does not set ✅ Check 13 (Draft precision): PASS — the draft is the embedded MTP head of ✅ Check 14 (Pareto coverage): PASS — one curve, ✅ Check 15 (PR description): PASS — the body's claims (1/2/3 PP4 prefill with a DEP16 decode, MTP6, HiCache, decode splits 1, decode Assessed commit: |
cquil11
approved these changes
Oct 9, 2026
cquil11
left a comment
cquil11
left a comment
Collaborator
There was a problem hiding this comment.
Approved for merge with the existing CODEOWNER sign-off and sweep reuse. Audited all eight throughput points in https://github.com/SemiAnalysisAI/InferenceX/actions/runs/37833971361: the actual AIPerf profiles use the 256k dataset, and profiling server-log AL interval means closely match the synthetic targets (4.41 or 4.80).
adibarra
added a commit
that referenced
this pull request
Oct 9, 2026
…s from the pareto bundles #3840's midcurve entries rerun two disaggregated and three aggregated points of the pareto entries by naming those points' standalone recipes, which the bundles replace. Their rows select the matching pareto overrides, so each still runs exactly the pareto point's recipe rather than a copy that could drift; the DEP16 points keep disagg-dep16-mtp6-variants.yaml. RECIPES.md notes that an entry rerunning another entry's points selects its overrides.
adibarra
added a commit
that referenced
this pull request
Oct 14, 2026
…t-recipe (#3805) * refactor(matrix): select srt-slurm recipes with srt-recipe-dir and srt-recipe Master configs name each row's recipe with srt-recipe (and eval-srt-recipe) relative to a per-entry srt-recipe-dir instead of CONFIG_FILE and EVAL_CONFIG_FILE additional-settings. Validation requires them on every srt-slurm row, checks the file and selected variant exist at load time, and rejects the old settings; llmd-vllm entries keep CONFIG_FILE. The generator emits project-relative paths, counts nodes from the row's recipe, and the launcher reads SRT_RECIPE/EVAL_SRT_RECIPE, staging the same recipes/<path>:<selector> argument for srtctl. Also drops the unread SYNTHETIC_ACCEPTANCE* and SLURM_PARTITION settings. * chore(recipes): drop stale CONFIG_FILE header comments Master rows now select bundle variants with srt-recipe, so the recipe headers no longer describe how to pick one. * fix(matrix): resolve srt recipes under the loaded config's project Recipe existence, selector checks and recipe node counts read files from the project that holds the loaded configs/ (or the root a caller passes) instead of the tooling checkout's repository_root(). The generator CLI and planner pass config_root(config files), Klaud live families pass the checkout root, and Klaud's canonical matrix fetches the candidate's recipes from the candidate head. * test(operatorx): give the generator fixture a project-layout srt recipe Master configs now name an srt-recipe resolved under the project that holds configs/, so the fixture places its master in configs/ beside a recipe. * fix(configs): select the DSv4.1-Flash ATOM AgentX recipe with srt-recipe-dir srt-recipe paths, which validation rejects now that every srt-slurm entry needs srt-recipe-dir. The entry sets srt-recipe-dir and its rows select agentic.yaml; the generated rows are unchanged. * chore(recipes): drop stale CONFIG_FILE headers from the DSv4.1-Flash GB300 AgentX recipes #3598 added the GB300 Dynamo+SGLang recipes with headers that say to select a variant with CONFIG_FILE. Master rows select them with srt-recipe now, so the headers stop at the variant layout like the other AgentX bundles. * refactor(matrix): deslop srt-recipe selection Set an AgentX row's srt-recipe fields once after the topology branch, and fetch each recipe file a Klaud family references once rather than once per row. * test: deslop srt-recipe selection tests The installed-package test no longer checks recipe node counts, which take an explicit root since srt-recipe and so no longer exercise caller-repository resolution. Drop unused monkeypatch fixtures and a history comment. * docs: deslop srt-recipe selection guidance Show srt-recipe in the CONFIGS.md search-space example, shorten its selection note, name srt-slurm recipes in the revision snapshot description, and shorten the debug-runs recipe path. * fix(configs): select each GB200 DSR1 FP8 low-latency variant explicitly The low-latency row named disagg-tp8-stp-variants.yaml without a selector, so one job submitted all three zip variants under a 4-node, 1-decode-worker label. Each variant now runs only the concurrencies it is tuned for, with its selector and its 7 TP8 decode workers on 16 nodes: zip_override_lowlat[0] (c8) at 4 and 8, zip_override_lowlat[1] (c16-32) at 16. zip_override_lowlat[2] is tuned for concurrency 64, which the entry never ran, so it has no row. * fix(configs): select the Qwen3.8 and Qwen3.5 GB300 AgentX recipes with srt-recipe-dir #3816 and #3840 added qwen3.8next-fp4-gb300-sglang-agentic-mtp and the Qwen3.5 FP8 GB300 midcurve entries with a project-relative srt-recipe path and CONFIG_FILE settings, which validation rejects now that every srt-slurm entry needs srt-recipe-dir. The entries set srt-recipe-dir and their rows select the same recipes and DEP16 overrides; the generated rows are unchanged. * fix(matrix): report a malformed srt-recipe as a validation error A recipe that fails to parse raised a raw yaml.YAMLError out of validate_master_config, so Klaud's canonical_matrix crashed on a malformed candidate recipe instead of rejecting it like other bad srt-recipe references. It now raises the same ValueError, naming the recipe file. * fix(configs): select the Qwen3.5 GB200 FP4 V2 MTP recipes with srt-recipe-dir #2594 added qwen3.5-fp4-gb200-dynamo-sglang-mtp with CONFIG_FILE settings, which validation rejects now that every srt-slurm entry needs srt-recipe-dir. The entry sets srt-recipe-dir and each row selects the same recipe; the generated rows are unchanged.
adibarra
added a commit
that referenced
this pull request
Oct 14, 2026
…r row (#3818) * refactor(srt): compose AgentX recipes from fragments and set power on master rows AgentX recipes, single- and multi-node, become fragments like the fixed-sequence ones. configs/srt-recipes/agentic-{single,multi}.yaml supply the client command; the binder writes the master model, image, precision and identity, and the launcher hands multi-node clients RESULT_DIR, AIPERF_DATASET_MMAP_CACHE_DIR and HF_HUB_CACHE from the volumes it mounts (agentic-volume-mounts replaces lanes._AGENTIC_CACHES). Dead keys and copies of the workflow environment leave the fragments; a fragment that sets a bound key fails before submission. A multinode search-space row with power: true replaces recipe telemetry blocks: the shared telemetry-dcgm.yaml is merged with the cluster's power-exporter-port and benchmark.concurrencies is bound after variant selection, so bundles can measure power. POWER_LANES now judges the row's field instead of reading recipe text, and inject_concurrencies is gone. Selected zip variants drop null-valued keys as srtctl apply does, and h200-dgxc GLM-5.2 FP8 AgentX keeps serving the Hub snapshot. * feat(srt): generate the bound recipes of AgentX points too infx generate now binds every srt-slurm point of the selected keys, AgentX as well as fixed-sequence, single- and multi-node, through the binder the launcher runs. The workflow environment it reproduces gains the AgentX inputs (IS_AGENTIC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, THINKING_MODE) and the multi-node request fields. A multi-node point takes the binder inputs the launcher derives from the one cluster its runner label schedules on: launch_path, the lane's POWER_LANES decision and the cluster's power-exporter-port give the DCGM telemetry block and bound concurrencies, and config.agentic_client_env, which no longer needs an SrtRun, gives the AgentX client paths. Such a point fails when its label spans several clusters or the lane refuses its power. The manifest records the cluster. * refactor(srt): compose the DSv4.1-Flash ATOM AgentX recipe from a fragment so binding it fails on its model block and benchmark MODEL. Like the other AgentX fragments it drops them, the client command its shared block supplies, and WEKA_LOADER_OVERRIDE, which names the corpus dsv41flash replays by default. Every point binds to main's selected variant without that override. * refactor(srt): deslop AgentX composition The multi-node binder reads IS_AGENTIC instead of defaulting a missing value to fixed-sequence, model_paths drops a guard the binder makes redundant, and RECIPES.md points at the bound-key tables instead of copying them. * refactor(srt): drop checks and env placeholders the binder makes dead select_recipe binds each variant before validate_recipe sees it, and the binder writes model.path, model.container, model.precision and benchmark.env.MODEL from the same environment, so those four comparisons can never fail. infx generate set KV_OFFLOADING and TOTAL_CPU_DRAM_GB to empty or zero placeholders for fixed-sequence points that nothing reads; it now sets them only for AgentX points. * refactor(srt): share the multi-node binder inputs with infx generate The launcher and infx generate each derived the DCGM exporter port and the AgentX client paths from the cluster, lane, request and power decision, and infx generate must match the launcher. Both now call config.binder_inputs. infx generate imports the launcher modules at the top: they import nothing that imports it back. * test(srt): drop binder tests that echo their inputs or repeat launcher tests The bound AgentX client test asserted values it passed in, and the CLI test's power and client-env tail repeated the srt driver tests that run the same module with those flags. The fragment-rejection test no longer pins the explanatory suffix, and an unused constant goes. * docs(srt): condense infx generate guidance RECIPES.md states the infx generate binder inputs in fewer words and drops a sentence the power-row table already makes. CONFIGS.md puts mounts back beside volume-mounts, which it describes, and the GLM-5.2 GB200 power comment moves onto the power field. * docs: drop recipe image and telemetry rules the binder now owns Recipes may no longer set model.container, identity.container.image or telemetry.enabled, yet the Klaud prompt and docs still asked for matching images, nuke.md still ran a script that rewrites model.container (it now fails on every recipe), and the power docs still gated energy collection on recipe telemetry. They now say the binder writes the master image and a power: true master row enables DCGM telemetry. * refactor(srt): compose main's new GB300 and B300 AgentX recipes from fragments #3816, #3840 and #3744 added full recipes: the Qwen3.8 GB300 and Qwen3.5 GB300 DEP16 recipes carry model blocks, the client command, benchmark MODEL and the client paths the launcher hands multi-node points, and both new recipes and the DSv4.1-Flash B300 one set the identity repo and image the binder writes. Like the other AgentX fragments they drop them, and WEKA_LOADER_OVERRIDE, which names both models' default corpus. The Qwen3.8 recipe keeps the TOTAL_CPU_DRAM_GB and KV_OFFLOADING the launcher checks against the point. Each point binds to main's selected variant, with the Qwen3.5 model alias bound as its Hub id like the other GB300 Qwen3.5 AgentX points. * refactor(srt): share one agentic.yaml block between the AgentX lanes agentic-single.yaml and agentic-multi.yaml were byte-identical. Single- and multi-node AgentX points now compose with configs/srt-recipes/agentic.yaml; the bound recipes are unchanged. * test(srt): reject a single-node variant sized for another DRAM budget The launcher rejects a variant whose TOTAL_CPU_DRAM_GB differs from the point's, the check that ties a recipe's literal host pools to the row's dram-utilization. No test covered it. * fix(srt): bind the AgentX client's IS_MULTINODE and power switch Aggregated multi-node AgentX recipes pinned IS_MULTINODE=false so the client would not measure power, and so published a single-node result shape for jobs that span nodes (the DSv4 GB300 TP8 agg point runs on two). The client's conc_N naming had to accept the workflow's CONC_LIST to keep collecting them. The Qwen3.5 B300 disagg recipe pinned ENABLE_AGENTX_POWER=0 instead. The binder now writes both into every multi-node AgentX recipe from the launcher's own state: IS_MULTINODE from the request, and ENABLE_AGENTX_POWER=1 exactly when the power decision collects AgentX or adapter power. A fragment that sets either fails composition. The pins are deleted, and so are the TP, PP_SIZE, PCP_SIZE and EP_SIZE copies: only single-node points read them, and multi-node results take their topology from the workflow's PREFILL_* and DECODE_* env. The client names results per concurrency on IS_MULTINODE alone. The DSv4 GB300 vLLM 1P4D recipe also pinned RUNNER_TYPE=gb300 and PREFILL_TP=1. Its prefill worker is DEP4 (tensor-parallel-size 1, data-parallel-size 4, expert parallel) on four GPUs, which the master row records as tp 4, ep 4, dp-attn like its DEP4 siblings; the pin published prefill_tp 1 and counted one prefill GPU instead of four. Those pins and the unread OFFLOADING copy are deleted, so the row publishes its runner label and topology like its siblings. The client's power mode, result directory and file name are unchanged for every point, except that the 192 multi-node points without power switch from an invalid power verdict (multinode_power_contract_missing) to none. The 35 formerly pinned points publish is_multinode true with prefill and decode topology. * refactor(srt): stop pinning workflow env in recipe fragments Fragments still set values the workflow exports from the matrix point or from runtime_settings.sh. The Kimi-K3 GB200 and GB300 fragments pinned REQUIRE_POWER=1, two TensorRT-LLM fragments pinned MAX_MODEL_LEN, MiniMax-M3 GB200 TRT copied KV_OFFLOAD_BACKEND_METADATA, the DSv4 GB300 frontends copied MODEL, MODEL_PREFIX, FRAMEWORK, PRECISION, DURATION, KV_OFFLOADING and CONC, and GLM-5.3 TileRT set MODEL=/model, which it published as the model. The binder now writes REQUIRE_POWER into every multi-node AgentX recipe: 1 when the workflow requires power or the lane's power rule does. The gb200-nv and gb300-nv Kimi-K3 AgentX rules require it, which keeps the pinned points strict and no others. BOUND_ENV rejects every name the workflow exports from the matrix point. MAX_MODEL_LEN goes (the AgentX client never reads it; evals take the job's), and so do the frontend copies, which the Dynamo frontend never reads. GLM-5.3 names its local checkpoint as the client's TOKENIZER, which the replay reads before MODEL, so it publishes zai-org/GLM-5.3 and keeps its tokenizer. Copies that equal the workflow's value go too: VLLM_ENGINE_READY_TIMEOUT_S 3600, PYTHONDONTWRITEBYTECODE, AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS 3600 and the MORI, NCCL and SGLang settings runtime_settings.sh exports for sglang-disagg. Every process keeps its effective value through the job environment. Fragments keep values that override the workflow's on purpose: the Dynamo routing and session settings, longer engine-ready timeouts, MORI_IO_QP_MAX_SEND_WR 32767 and single-node variant labels.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add three Qwen3.5-397B-A17B-FP8 GB300 AgentX configurations pairing one, two, or three PP4 prefill workers with one TP/DP/EP16 decode worker. The shared recipe bundle uses MTP6, HiCache, decode sequence-length splits 1, and
max-mamba-cache-size: 1872for decode.Two matrix entries select five disaggregated configurations and three existing aggregate configurations on
lmsysorg/sglang:nightly-dev-cu13-20260913-fa663e72, retaining Dynamo revisionad6f7d54ce7b99c53576b10482755c93c921fbac. PP4 prefill roles setPYTHONOPTIMIZE=1, which disables Python assertions, to allow MTP with overlap scheduling on this nightly; decode and aggregate roles retain normal assertion behavior.中文
新增三种 Qwen3.5-397B-A17B-FP8 GB300 AgentX 配置，分别将一、二、三个 PP4 prefill worker 与一个 TP/DP/EP16 decode worker 配对。共享配方使用 MTP6、HiCache、decode sequence-length splits 1，并将 decode 的
max-mamba-cache-size设为 1872。两个 matrix 条目选择五种分离式配置和三种已有聚合配置，使用
lmsysorg/sglang:nightly-dev-cu13-20260913-fa663e72，并沿用 Dynamo 固定版本ad6f7d54ce7b99c53576b10482755c93c921fbac。PP4 prefill 角色设置PYTHONOPTIMIZE=1，禁用 Python 断言，以便在此 nightly 中同时使用 MTP 和 overlap scheduling；decode 和聚合角色保留正常的断言行为。AI model disclosure
gpt-6.1-sol: prepared the recipe settings and matrix configuration.claude-opus-5-5: reviewed the recipe settings and selected configurations.AI 模型说明：
gpt-6.1-sol编写配方参数和 matrix 配置；claude-opus-5-5审阅配方参数及所选配置；Codex 整理配方变体、准备配置及变更记录、执行本地检查并起草本说明，当前精确模型及版本无法核实。