[AgentX] DeepSeek-V4-Pro-0813 llm-d on GB200 - #3741

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

… the matrix point (#3809)

* refactor(srt): compose fixed-sequence recipes from fragments and bind the point

Active fixed-sequence srt-slurm recipes keep only recipe-specific settings.
At launch the lane's shared block (configs/srt-recipes/) is merged under the
fragment, the row's variant is selected, and the binder writes the matrix
point: model path, image, precision, ISL/OSL, single-node client values
(MODEL, CONC, RANDOM_RANGE_RATIO, USE_CHAT_TEMPLATE) and telemetry
concurrencies. A fragment that sets a bound key fails before submission.

The hf:<MODEL> model path maps to the cluster's staged checkpoint and the
master image to its staged container, so recipes stop naming aliases. The
mi355x-amds DeepSeek-R1 MXFP4 checkpoint moves into models.entries. Six
b300-fp8 trtllm recipes declared fp4 precision; it is now bound to fp8.

* feat(srt): add infx generate for bound fixed-sequence recipes

`infx generate --config-key <key|glob> --output-dir <dir>` writes the bound
recipe of every fixed-sequence point and variant, validated by the pinned
srtctl, with a manifest mapping files to matrix points. srtctl's import
dependencies sit behind the new `recipes` extra. RECIPES.md documents
fragments, composition and the command.

* refactor(srt): set the multi-node fixed-sequence TOKENIZER in the shared block

Every active multi-node fixed-sequence fragment repeated
benchmark.env.TOKENIZER: /model. The shared block
configs/srt-recipes/fixed-sequence-multi.yaml now sets it and the fragments
drop it, along with the benchmark blocks it was the only entry of.

The three mi355x fragments that lacked it (DeepSeek-R1 FP8 disagg variants,
Qwen3.5 FP4 and FP8 1p1d) now tokenize with the mounted checkpoint too
instead of the served model id. Every other bound recipe is unchanged.

* fix(srt): bind exactly one multi-node variant in infx generate

infx generate bound every variant a multi-node selector matched, while the launcher's binder refuses anything but one, so it wrote recipes for points that launch rejects, such as dsr1-fp8-gb200-dynamo-sglang. Both now share bind_multinode, and generate fails such a point as launch does.

* refactor(srt): deslop the fixed-sequence binder

The binder parses CONC_LIST with the launcher's parse_concurrencies, now in workload.py, so telemetry concurrencies get its canonical and unique checks. Define the bound recipe file name once, drop generate's MODEL_PREFIX and RUN_EVAL inputs that nothing on its path reads, name what _validated does, and drop a model guard the binder makes redundant. Fragment checks drop two cases that repeat a covered branch.

* test(srt): drop assertions on where the bound recipe lives

The content checks that follow already prove srtctl ran the bound variant.

* docs: deslop fixed-sequence fragment guidance

Drop restatements of the shared blocks and alias history from RECIPES.md, and correct which keys add-model-hardware says fragments omit.

* build: drop test pins the recipes extra already declares

CI and the testing guide install every extra with the test group.

* fix(srt): fail fast when a fixed-sequence point's cluster stages no checkpoint

Fragments no longer name model.path aliases, so the alias guard never fired
for fixed-sequence recipes: an unstaged MODEL left model_paths empty and
srtctl resolved the bound hf:<MODEL> from the Hub on the compute nodes.
model_paths now raises before submission instead, unless an Override row
sets hub to load the Hub snapshot. The mi355x-amds DeepSeek-R1 FP8 and
Qwen3.5 FP4 SGLang disagg lanes, whose fragments named hf: paths before
they were bound, set it and keep loading the Hub snapshot.

* refactor(srt): compose the Qwen3.5 GB200 FP4 V2 MTP recipes from fragments

#2594 added nine standalone recipes for qwen3.5-fp4-gb200-dynamo-sglang-mtp
that name a model alias, the image and the precision, and run srt-slurm's
sa-bench. They become fragments: the binder writes the master model, image
and precision, and the shared fixed-sequence-multi.yaml block runs the
InferenceX client, as every other multi-node fixed-sequence recipe has since
#3415. The client keeps sa-bench's settings: the point's lengths, the
workflow's 0.8 range ratio, ten prompts and two warmups per concurrency, an
unthrottled request rate, the chat template, ignore-EOS and the /model
tokenizer. The c12288 point's sa-bench reuse_http_connections has no
counterpart; the client opens a connection per request, as it does for every
other point.

…r row (#3818)

* refactor(srt): compose AgentX recipes from fragments and set power on master rows

AgentX recipes, single- and multi-node, become fragments like the
fixed-sequence ones. configs/srt-recipes/agentic-{single,multi}.yaml supply
the client command; the binder writes the master model, image, precision and
identity, and the launcher hands multi-node clients RESULT_DIR,
AIPERF_DATASET_MMAP_CACHE_DIR and HF_HUB_CACHE from the volumes it mounts
(agentic-volume-mounts replaces lanes._AGENTIC_CACHES). Dead keys and copies
of the workflow environment leave the fragments; a fragment that sets a bound
key fails before submission.

A multinode search-space row with power: true replaces recipe telemetry
blocks: the shared telemetry-dcgm.yaml is merged with the cluster's
power-exporter-port and benchmark.concurrencies is bound after variant
selection, so bundles can measure power. POWER_LANES now judges the row's
field instead of reading recipe text, and inject_concurrencies is gone.

Selected zip variants drop null-valued keys as srtctl apply does, and
h200-dgxc GLM-5.2 FP8 AgentX keeps serving the Hub snapshot.

* feat(srt): generate the bound recipes of AgentX points too

infx generate now binds every srt-slurm point of the selected keys, AgentX
as well as fixed-sequence, single- and multi-node, through the binder the
launcher runs. The workflow environment it reproduces gains the AgentX
inputs (IS_AGENTIC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, THINKING_MODE) and the
multi-node request fields.

A multi-node point takes the binder inputs the launcher derives from the one
cluster its runner label schedules on: launch_path, the lane's POWER_LANES
decision and the cluster's power-exporter-port give the DCGM telemetry block
and bound concurrencies, and config.agentic_client_env, which no longer needs
an SrtRun, gives the AgentX client paths. Such a point fails when its label
spans several clusters or the lane refuses its power. The manifest records
the cluster.

* refactor(srt): compose the DSv4.1-Flash ATOM AgentX recipe from a fragment

so binding it fails on its model block and benchmark MODEL. Like the other
AgentX fragments it drops them, the client command its shared block supplies,
and WEKA_LOADER_OVERRIDE, which names the corpus dsv41flash replays by
default. Every point binds to main's selected variant without that override.

* refactor(srt): deslop AgentX composition

The multi-node binder reads IS_AGENTIC instead of defaulting a missing value to fixed-sequence, model_paths drops a guard the binder makes redundant, and RECIPES.md points at the bound-key tables instead of copying them.

* refactor(srt): drop checks and env placeholders the binder makes dead

select_recipe binds each variant before validate_recipe sees it, and the
binder writes model.path, model.container, model.precision and
benchmark.env.MODEL from the same environment, so those four comparisons
can never fail. infx generate set KV_OFFLOADING and TOTAL_CPU_DRAM_GB to
empty or zero placeholders for fixed-sequence points that nothing reads;
it now sets them only for AgentX points.

* refactor(srt): share the multi-node binder inputs with infx generate

The launcher and infx generate each derived the DCGM exporter port and
the AgentX client paths from the cluster, lane, request and power
decision, and infx generate must match the launcher. Both now call
config.binder_inputs. infx generate imports the launcher modules at the
top: they import nothing that imports it back.

* test(srt): drop binder tests that echo their inputs or repeat launcher tests

The bound AgentX client test asserted values it passed in, and the CLI
test's power and client-env tail repeated the srt driver tests that run the
same module with those flags. The fragment-rejection test no longer pins
the explanatory suffix, and an unused constant goes.

* docs(srt): condense infx generate guidance

RECIPES.md states the infx generate binder inputs in fewer words and
drops a sentence the power-row table already makes. CONFIGS.md puts mounts
back beside volume-mounts, which it describes, and the GLM-5.2 GB200 power
comment moves onto the power field.

* docs: drop recipe image and telemetry rules the binder now owns

Recipes may no longer set model.container, identity.container.image or
telemetry.enabled, yet the Klaud prompt and docs still asked for matching
images, nuke.md still ran a script that rewrites model.container (it now
fails on every recipe), and the power docs still gated energy collection
on recipe telemetry. They now say the binder writes the master image and
a power: true master row enables DCGM telemetry.

* refactor(srt): compose main's new GB300 and B300 AgentX recipes from fragments

#3816, #3840 and #3744 added full recipes: the Qwen3.8 GB300 and Qwen3.5
GB300 DEP16 recipes carry model blocks, the client command, benchmark MODEL
and the client paths the launcher hands multi-node points, and both new
recipes and the DSv4.1-Flash B300 one set the identity repo and image the
binder writes. Like the other AgentX fragments they drop them, and
WEKA_LOADER_OVERRIDE, which names both models' default corpus. The Qwen3.8
recipe keeps the TOTAL_CPU_DRAM_GB and KV_OFFLOADING the launcher checks
against the point.

Each point binds to main's selected variant, with the Qwen3.5 model alias
bound as its Hub id like the other GB300 Qwen3.5 AgentX points.

* refactor(srt): share one agentic.yaml block between the AgentX lanes

agentic-single.yaml and agentic-multi.yaml were byte-identical. Single- and multi-node AgentX points now compose with configs/srt-recipes/agentic.yaml; the bound recipes are unchanged.

* test(srt): reject a single-node variant sized for another DRAM budget

The launcher rejects a variant whose TOTAL_CPU_DRAM_GB differs from the
point's, the check that ties a recipe's literal host pools to the row's
dram-utilization. No test covered it.

* fix(srt): bind the AgentX client's IS_MULTINODE and power switch

Aggregated multi-node AgentX recipes pinned IS_MULTINODE=false so the
client would not measure power, and so published a single-node result
shape for jobs that span nodes (the DSv4 GB300 TP8 agg point runs on
two). The client's conc_N naming had to accept the workflow's CONC_LIST
to keep collecting them. The Qwen3.5 B300 disagg recipe pinned
ENABLE_AGENTX_POWER=0 instead.

The binder now writes both into every multi-node AgentX recipe from the
launcher's own state: IS_MULTINODE from the request, and
ENABLE_AGENTX_POWER=1 exactly when the power decision collects AgentX or
adapter power. A fragment that sets either fails composition. The pins
are deleted, and so are the TP, PP_SIZE, PCP_SIZE and EP_SIZE copies:
only single-node points read them, and multi-node results take their
topology from the workflow's PREFILL_* and DECODE_* env. The client
names results per concurrency on IS_MULTINODE alone.

The DSv4 GB300 vLLM 1P4D recipe also pinned RUNNER_TYPE=gb300 and
PREFILL_TP=1. Its prefill worker is DEP4 (tensor-parallel-size 1,
data-parallel-size 4, expert parallel) on four GPUs, which the master
row records as tp 4, ep 4, dp-attn like its DEP4 siblings; the pin
published prefill_tp 1 and counted one prefill GPU instead of four.
Those pins and the unread OFFLOADING copy are deleted, so the row
publishes its runner label and topology like its siblings.

The client's power mode, result directory and file name are unchanged
for every point, except that the 192 multi-node points without power
switch from an invalid power verdict (multinode_power_contract_missing)
to none. The 35 formerly pinned points publish is_multinode true with
prefill and decode topology.

* refactor(srt): stop pinning workflow env in recipe fragments

Fragments still set values the workflow exports from the matrix point or
from runtime_settings.sh. The Kimi-K3 GB200 and GB300 fragments pinned
REQUIRE_POWER=1, two TensorRT-LLM fragments pinned MAX_MODEL_LEN, MiniMax-M3
GB200 TRT copied KV_OFFLOAD_BACKEND_METADATA, the DSv4 GB300 frontends copied
MODEL, MODEL_PREFIX, FRAMEWORK, PRECISION, DURATION, KV_OFFLOADING and CONC,
and GLM-5.3 TileRT set MODEL=/model, which it published as the model.

The binder now writes REQUIRE_POWER into every multi-node AgentX recipe: 1
when the workflow requires power or the lane's power rule does. The
gb200-nv and gb300-nv Kimi-K3 AgentX rules require it, which keeps the
pinned points strict and no others. BOUND_ENV rejects every name the
workflow exports from the matrix point. MAX_MODEL_LEN goes (the AgentX
client never reads it; evals take the job's), and so do the frontend
copies, which the Dynamo frontend never reads. GLM-5.3 names its local
checkpoint as the client's TOKENIZER, which the replay reads before MODEL,
so it publishes zai-org/GLM-5.3 and keeps its tokenizer.

Copies that equal the workflow's value go too: VLLM_ENGINE_READY_TIMEOUT_S
3600, PYTHONDONTWRITEBYTECODE, AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS 3600
and the MORI, NCCL and SGLang settings runtime_settings.sh exports for
sglang-disagg. Every process keeps its effective value through the job
environment. Fragments keep values that override the workflow's on
purpose: the Dynamo routing and session settings, longer engine-ready
timeouts, MORI_IO_QP_MAX_SEND_WR 32767 and single-node variant labels.

…affected config keys (#3812)

* feat(matrix): fingerprint the concrete recipe each benchmark row runs

recipe-fingerprint hashed only the generated matrix row, so editing a
fragment, a shared block or an AgentX variant left it unchanged. It now
hashes the row's fields (all but conc, exp-name and the fingerprint) with
the one variant the launcher submits for the row: the fragment composed
with its shared block (and the telemetry block for a power row), the
selected variant and the bound image, model, precision and, for fixed
sequences, lengths. Concurrency values and the job name stay out, and the
cluster facts the launcher adds (staged paths, mounts, client cache paths,
exporter port, fabric) never enter it, so one recipe keeps one fingerprint
across concurrencies and clusters. An eval-srt-recipe contributes only its
path. Rows without an srt-slurm recipe keep the row hash.

The planner runs without srtctl, so srt_slurm/variants.py reproduces
srtctl's override-bundle expansion and the planner reuses the launcher's
variant selection and binding. A row that no variant serves now fails
planning instead of its launch.

Klaud matches published fingerprints when it freezes a baseline. Producer
revisions now recompute theirs with their own infx.matrix.fingerprint, or
the row hash when they predate it, and the canonical family fetches the
head's shared blocks with its recipes.

* feat(matrix): list the config keys a change affects

`python -m infx.matrix.changed --base <ref>` generates the base matrix with
the base revision's own generator, fingerprints both sides under their own
recipes and prints every config key whose points (recipe fingerprint and
concurrency) were added or removed, with point counts. --json prints the
same report for tools. Authors use it to decide which keys a perf-changelog
entry should rerun.

* test(matrix): drop assertions other tests already make

test_expansion_matches_srtctl already compares the zip, override and unselected expansions in full against srtctl, and the change report's added and removed concurrencies already prove the edited fragment's fingerprints changed.

* refactor(matrix): deslop recipe fingerprints

variants.py owns without_nulls and the zip selector pattern that synthetic_acceptance duplicated. bound_variant no longer gates power_port on the row's power, which only the fingerprint needed; the fingerprint passes its stand-in port for power rows itself. Drop the CONC_LIST exclusion that no bound recipe can carry, the listed flag in producer regeneration, and defensive exp-name lookups, and trim narrating docstrings.

* docs: condense recipe fingerprint guidance

* test(fingerprint): edit the shared agentic.yaml block in the AgentX case

… refs (#3820)

* refactor(srt): take cluster fabric facts from runners.yaml

Recipes stop hardcoding the cluster's network hardware. Each srt-slurm
cluster profile gains a typed fabric record: ucx-net-devices, nccl-ib-hca,
ib-devices, socket-ifname and mooncake-devices are lists, and
mooncake-gid-index, mori-rdma-tc and mori-io-tc are numbers. A recipe value
'@fabric.<name>' takes the job cluster's field after binding, lists
comma-joined and numbers as decimal strings. The launcher hands both
binders the cluster's rendered fabric; an unknown name, a reference inside
a longer string, or a field the cluster does not set fails before
submission. Nothing resolves references at plan time.

Literals equal to their cluster's fact become references in 72 active
recipes on gb200-nv, gb300-nv, b200-nscale, b300-dsxe, h200-dgxc and
mi355x-amds. Deliberate per-recipe values stay literal: the MI355X UMBP
recipe's permuted rail order, B200 Mooncake's four of eight HCAs, and empty
Mooncake device lists that leave discovery to Mooncake.

infx generate resolves references for a point whose runner label names one
cluster and records that cluster in the manifest; otherwise references
stay and the manifest's cluster is null.

* feat(srt): check recipe fabric references at matrix generation

Matrix generation now fails a row whose selected recipe variant (srt-recipe
or eval-srt-recipe) has an '@fabric.<name>' value naming no Fabric field, or
a field that a cluster the row's runner label reaches does not set. The
launcher still resolves references at bind time and fingerprints hash them
as written.

Cluster env and srt-slurm.host-setup.env values may reference the cluster's
own fabric the same way and resolve when the record loads; env keeps
rejecting commas after resolution. mi355x-amds takes MORI_RDMA_TC and
IBDEVICES from its fabric instead of repeating them.

* chore: reconcile srt-fabric-refs with lower branch after linearizing

* fix(srt): keep the fabric reference constant importable on Python 3.10

The binder runs in srtctl's Python 3.10 venv, where importing infx.clusters fails on typing.Self.

* refactor(srt): merge RDMA NIC fabric fields into rdma-devices

nccl-ib-hca, mooncake-devices and ib-devices held the same RDMA HCA list on
every cluster that set them. One Fabric field, rdma-devices, now carries that
list for NCCL_IB_HCA, the Mooncake device_name, SGLang
disaggregation-ib-device and IBDEVICES, and each cluster lists its HCAs once.

Bound recipes are unchanged. Rows whose recipes reference the field get new
fingerprints, since references are hashed as written.

* refactor(srt): take the GB300 DSv4 Mooncake RDMA devices from the fabric

prefill role. Like the other recipes' NIC lists, both read '@fabric.rdma-devices',
which renders the same mlx5_0..mlx5_3 list on gb300-nv.

* test(srt): drop fabric assertions other tests already make

The workload tests already prove the multi-node binder resolves --fabric, the single-node driver case proves the launcher passes the job cluster's rendering, and the matrix tests already cover an unknown fabric name.

* refactor(srt): fail fabric lookups for clusters without a record

infx generate now raises the binder inputs' missing srt-slurm settings error instead of binding an empty fabric, and the matrix check reads the record a cluster label names instead of treating a missing one as an empty fabric.

* docs: condense fabric reference guidance

* refactor(srt): take the GB200 DSv4 Mooncake RDMA devices from the fabric

the decode stores and the prefill role. Like the GB300 recipes, both read
'@fabric.rdma-devices', which renders the same mlx5_0..mlx5_3 list on
gb200-nv.

* refactor(srt): resolve every fabric reference with resolve_fabric

Cluster env and host-setup.env values and the matrix plan-time check now
call the binder's resolve_fabric instead of Fabric.field_name,
Fabric.resolve and a second string walk. An unset field reads "this
cluster sets no srt-slurm.fabric.<name>" everywhere, and the matrix error
names the recipe and the first cluster that fails. A runner that no
cluster label lists, such as gb300-nv, still has its reference names
checked. A cluster env: null is now rejected instead of read as an empty
mapping.

The embedded-reference generation case is dropped: it ran the same
resolve_fabric branch as the binder's IBDEVICES case.

* refactor(srt): name fabric fields after cluster RDMA facts

Fabric fields now name what the cluster has rather than the software
that reads it: ucx-net-devices becomes rdma-device-ports,
mooncake-gid-index becomes rdma-gid-index, and mori-rdma-tc and
mori-io-tc, both 104 on mi355x-amds, become one rdma-tc. Recipes and the
mi355x-amds cluster env keep their engine variable names (UCX_NET_DEVICES,
MC_GID_INDEX, MORI_RDMA_TC, MORI_IO_TC) and point them at the new
references, so every bound recipe renders the same values.

…er config (#3821)

* refactor(srt): bind script-installed component versions from the master config

Repo setup scripts that install a component now read its version from the
recipe instead of pinning it, and the binder writes the point's master
version after variant selection, so the submitted recipe shows what gets
installed:

- vllm-router.sh (was vllm-router-0.1.14.sh) reads ROUTER_VERSION.
- vllm-mooncake.sh (was vllm-mooncake-0.3.11.sh) reads
  KV_OFFLOAD_BACKEND_VERSION; kimik3-b300-mooncake.sh and the
  mooncake-master preambles of the Kimi-K3 B300 and MiniMax-M3 H100
  recipes run it instead of pinning their own pip install.
- lmcache-mp-rocm.sh reads KV_OFFLOAD_BACKEND_VERSION.
- glm5.3-tilert-rocm.sh reads ROUTER_VERSION, the tilert-pd-router
  release; the GLM-5.3 recipe drops TILERT_VERSION.

The binder writes the version into the top-level environment when
setup_script is one of these installers (workers and static routers) and
into the env of each service whose preamble runs one, since services do
not inherit environment. A point that does not declare the version fails
before submission, a vllm-router pin in SETUP_PIP_PACKAGES must match the
master router, and a fragment may not set the version variables.
The launcher and infx generate also fail a bound recipe whose setup_script
is in neither configs/ nor the srt-slurm checkout's, where srtctl itself
only warns.

* test(srt): drop duplicate component version tests

The router half of the binding test reruns the setup_script branch the Mooncake half covers, the pinned-router acceptance only echoed its input, and the setup-script check is called directly instead of through a copy of the multi-node CLI harness.

* refactor(srt): deslop component version binding

_fragment_versions yields its findings, the one-entry PIP_COMPONENTS registry becomes the vllm-router check it was, and lmcache-mp-rocm.sh reads KV_OFFLOAD_BACKEND_VERSION without an alias.

* docs: condense component version guidance

* refactor(srt): drop the binder's fragment path from component version errors

bind_workload took a required source only to prefix two component version
errors with the fragment path. No other binder error names the fragment,
and the planner's recipe fingerprint already prefixes the srt-recipe, so
those errors printed the path twice. check_setup_script keeps its source.

… config key (#3824)

* fix(launch): apply lane job edits to the bound recipe

prepare_recipe edited the staged fragment before the binder ran. Its text
edits only find a flat recipe's top-level name and six-space
watchdog-timeout, so bundle variants kept their recipe names as srtctl's
run name and never got the cluster's dist-timeout. The edits now apply to
the recipe the binder writes, which is flat for every variant. Slurm job
names were unaffected: srtctl names the job from RUNNER_NAME.

* refactor(recipes): fold per-point recipe families into one bundle per entry

Master-config entries whose rows each named their own standalone recipe now
select overrides of one <agg|disagg>[-axis]-variants.yaml: base holds what
the points share and each override_<topology>_c<conc> holds only its point's
deltas, including the job name. Ten AgentX entries (50 recipes) and
seventeen fixed-sequence entries (184 recipes) move into 27 bundles. Every
moved row's selected variant equals its old recipe after composition, with
and without the power telemetry block, and the full-sweep matrices differ
only in those rows' srt-recipe strings.

Entries whose points mix recipe designs, such as a TP low-latency recipe
beside wide-EP ones, keep their standalone files. Section-header comments,
commented-out settings and per-point labels that restate the topology are
not carried into the bundles. RECIPES.md describes the convention.

* refactor(recipes): fold the GLM-5.2 GB300 SGLang AgentX point recipes into bundles

the other AgentX families, the three aggregate points select overrides of
agg-variants.yaml and the three low-latency 1PxD points overrides of
disagg-variants.yaml; each override selects exactly its old recipe. The
DEP8/DEP16 2P1D point is a different design and keeps its own file.

* test(launch): drop recipe edit assertions other tests already make

The health-check floor and model path of a bound bundle variant are already covered by test_srt_config and the multi-node lane test, and the bundle-shaped health budget test exercises inputs that only reach raise_health_attempts as the binder's flat output.

* refactor(launch): inline recipe_relpath into recipe_mirror_path

Its other caller went away when the lane edits moved to the bound recipe.

* docs: drop the per-override name rule and tidy bundle comments

Every launch renames the bound recipe and fingerprints exclude name, so bundles need not set one per override. Drop a misindented commented-out flag and move the NATS comment above request_plane.

* refactor(recipes): select the Qwen3.5 GB300 FP8 AgentX midcurve points from the pareto bundles

of the pareto entries by naming those points' standalone recipes, which the
bundles replace. Their rows select the matching pareto overrides, so each
still runs exactly the pareto point's recipe rather than a copy that could
drift; the DEP16 points keep disagg-dep16-mtp6-variants.yaml. RECIPES.md
notes that an entry rerunning another entry's points selects its overrides.

* refactor(launch): edit the bound recipe as data

prepare_recipe now edits the binder's safe_dump output through one
safe_load/safe_dump round trip instead of regex and line edits that
depended on PyYAML's indentation: it sets the top-level name, raises
health_check.max_attempts to the floor, and adds dist-timeout to each
role whose args set watchdog-timeout. For all 589 bound multi-node
points of every master key and every multi-node recipe variant, with and
without a dist timeout, the written recipe parses identically to the old
edits. The helper tests fold into one prepare_recipe test.

* refactor(recipes): drop the per-override names from the folded bundles

The 29 bundles folded from per-point recipes carried each old file's
top-level name into its override, and some no longer matched the
topology. Nothing reads them: every launch renames the bound recipe to
the runner's job name, srtctl names the Slurm job from RUNNER_NAME, and
fingerprints exclude name. Variants now take srtctl's
<base name>_<override key> fallback, which only changes infx generate
output.

…idence (#4036)

明确参数化命令和适用官方文档可满足配方覆盖要求；将缺少草稿精度证据改为警告，保留已证实违规的失败判定，并同步清单说明和发布器回归测试。

Signed-off-by: Xin Li <xinli@nvidia.com>

…-K3 PD 配置 (#3582)

* feat(launch): support read-only Kimi-K3 assets

为 Kimi-K3 PD 声明模型资源与只读 draft/provider 挂载；保留现有可写缓存目录的创建行为，不改共享镜像准备和任务生命周期。

* feat(config): add native Kimi-K3 PD recipes

Add topology-named native variants with a digest-pinned official worker image, role-owned serving settings and default evaluation selection. Preserve upstream request-ID randomization and keep benchmark and cluster configuration separate.

新增按拓扑命名的原生配置变体，使用 digest 固定的官方 worker 镜像、按角色管理的 serving 参数及默认精度评测；遵循上游 request-ID 随机化，将 benchmark 和集群配置分层。

* fix(srt-slurm): backport discovery connector template binding

Carry NVIDIA/srt-slurm #508 through the existing patches directory at the shared submodule API. Exercise real job-local patch application and conflict handling, then run the upstream P/D command-rendering regressions in an isolated checkout. Remove the patch once the shared dependency includes it.

沿用既有 patches 目录回补 NVIDIA/srt-slurm #508，与共享 submodule API 保持一致。验证任务私有 checkout 的补丁应用和冲突退出，再在隔离进程中执行上游 P/D 命令渲染回归；共享依赖包含该修复后删除补丁。

* refactor: use native asset mounts for Kimi-K3 recipes

Declare draft checkpoint and Ionic provider mounts through existing recipe and cluster configuration. Keep shared launch code unchanged, carry only the #508 runtime backend fix, and record recipe details in YAML and the changelog.

中文：通过现有 recipe 和集群配置声明 draft checkpoint 与 Ionic provider 挂载。共享 launch 代码保持不变，仅携带 #508 的运行时 backend 修复，将 recipe 信息统一记录在 YAML 和 changelog。

* perf(config): tune and simplify Kimi-K3 PD recipe settings

Enable INT4 QuickReduce with a 256 MiB limit and FlyDSL stage-2 FP8 on both roles. Use default nonblocking waits and breakable-graph settings. Scope draft and Ionic mount paths to the recipe. Configure 1P1D c48/c56, 1P2D c24/c48 and 1P3D c12/c24 with prefill-only CPU offload and point-specific decode graph coverage. Update the existing performance changelog entry.

中文：在 P/D 两侧启用 INT4 QuickReduce、256 MiB 大小上限和 FlyDSL stage-2 FP8，使用默认非阻塞等待与 breakable graph 设置。将 draft 和 Ionic 挂载路径收束到 recipe。配置 1P1D c48/c56、1P2D c24/c48 和 1P3D c12/c24，使用仅 prefill 侧 CPU offload 与各点对应的 decode graph 覆盖，并更新当前 PR 的性能变更记录。

* fix(config): use default AgentX failure threshold for Kimi-K3 PD

Remove the recipe's live failure threshold override so all Kimi-K3 PD variants use the pinned AgentX scenario's 10% default. Keep the final result threshold, both roles' stream interval, and optional server-metrics validation unchanged.

移除 Kimi-K3 PD 配方的运行中错误率覆盖，所有变体沿用固定版本 AgentX 场景的默认 10% 门槛。最终结果校验门槛、P/D 流式输出间隔及可选 server-metrics 校验保持不变。

* fix(srt): clarify discovery patch scope

说明 #508 仅修复 srt-slurm 启动器的 discovery 配置渲染，不修改推理引擎、kernel、模型计算或镜像。补丁应用后的代码保持不变。

* fix(kimik3): migrate PD recipe to srt-recipe selection and AgentX fragments

Main now selects srt-slurm variants with srt-recipe-dir/srt-recipe and binds
model, image, precision, client env and concurrencies from the master row, so
the recipe drops those keys and the shared agentic.yaml supplies the client
command.

---------

Co-authored-by: Cam Quilici <cjquilici@gmail.com>
Co-authored-by: billishyahao <bill.he@amd.com>

* chore: remove comments from srt-slurm recipe YAML

删除单节点和多节点 srt-slurm 配方 YAML 中的注释，保持配置内容不变。

* chore: keep explanatory recipe comments

Restore inline comments that explain non-obvious settings (workarounds,
timeouts, memory sizing, bug references). Header blocks, section labels,
topology/provenance descriptions, perf result notes and commented-out
config stay removed.

---------

Co-authored-by: cquil11 <60715037+cquil11@users.noreply.github.com>

…/ 清理 infx 注释并保留指定目录 (#4039)

* chore: remove infx comments outside bench_serving

删除 infx 中 bench_serving 目录之外的说明性注释，保留功能性指令和代码行为。

* chore: preserve golden_al_distribution comments

按要求保留 golden_al_distribution 的全部注释和原始内容。

Ban header comments and historical comments; comments must give succinct
technical explanation of a specific line or block.

禁止文件头注释和描述代码历史演变的注释；注释只能简洁地从技术角度解释具体代码或配置。

…Agent 指南 (#4042)

* docs: remove redundant pipeline guides

删除三份重复说明文档并修复引用，保留 Claude 命令、技能、已知限制及其操作文档。

* docs: consolidate result and ingest recovery instructions

将结果检查和恢复流程合并到 Claude 恢复命令，删除重复指南并保留验证与权限边界。

* docs: remove requested guides and repair agent references

删除指定的说明文档，保留四项核心文档与全部 Claude 命令，将必要操作规则移入现有入口并修复引用。

从 AMD 主配置的 CODEOWNERS 条目中移除无效所有者 @yctseng0211，保留其余所有者。

…#3781)

* feat(power): bind native GPU telemetry into composed recipes

* fix(power): include native telemetry in revision recipe loading

* chore: remove PR 3781 perf changelog entry

* docs: trim AMD exporter notes in CONFIGS.md

* test(power): remove producer replay test and dependency

* refactor(power): bind single-node required power from REQUIRE_POWER

The launcher no longer re-reads the bound recipe, patches its request, or
writes $GITHUB_ENV. REQUIRE_POWER alone sets telemetry.required, so a
single-node fragment may not set it, and result processing takes the
producer SHA from the launcher stamp as the multi-node workflow does.

* refactor(power): gate single-node telemetry only when composing

select_recipe already omits the native telemetry block for eval-only points,
so the binder no longer rewrites telemetry.enabled.

* refactor(power): share the AgentX power adapter invocation

Single-node and multi-node AgentX run the adapter through one helper, and the
single-node adapter writes its sidecar into the workspace instead of the
launcher copying it there.

* refactor(power): derive single-node AgentX GPU count from the aggregate

The aggregate already records num_gpus from the same environment as
GPU_COUNT, so the adapter drops --expected-num-gpus.

* chore(runners): share the AMD exporter settings through YAML anchors

* ci(power): drop power-audit upload paths no job writes

---------

Co-authored-by: Cam Quilici <cjquilici@gmail.com>

…wer_model] 支持跨系统的实测 CPU socket 功耗输入 (#4031)

* feat: add measured x86 CPU socket input to power_model

中文：为 power_model 新增实测 x86 CPU socket 功耗输入，仅替代建模的 x86 CPU 功耗

* refactor: make x86 CPU socket validation explicit

中文：明确检查 x86 CPU socket，并删除重复的测试注释。

* feat: apply both measured socket inputs across system families

中文：让两种实测 socket 功耗输入适用于 HGX 和机架系统。

…urve / [NVIDIA][AgentX] 更新 GB200 DeepSeek-V4.1-Flash SGLang 聚合曲线 (#3927)

* perf(dsv41flash): refresh GB200 SGLang AgentX aggregate curve

Repin dsv41flash-fp4-gb200-sglang-agentic-dspark to
lmsysorg/sglang:nightly-dev-cu13-20261011-d32d8fa0 (multiarch digest
pinned) and replace the TP4/EP1 and TP2/EP1 C1-C128 sweeps with TP4/EP4
points at C1, C4, C8 and C16 and aggregate TP4/EP4 points with DP attention
at C32, C64, C128 and C192 behind sglang-router 0.3.2. The recipe carries
the per-point prefill-decode interval, prefill chunking, admission, memory
and SWA prefix-tail settings, and appends the matching changelog entry.

将 dsv41flash-fp4-gb200-sglang-agentic-dspark 更新为
lmsysorg/sglang:nightly-dev-cu13-20261011-d32d8fa0（固定 multiarch
digest），并以 TP4/EP4 的 C1、C4、C8、C16，以及在 sglang-router 0.3.2
之后启用 DP attention 的聚合式 TP4/EP4 的 C32、C64、C128、C192，取代原本的
TP4/EP1 与 TP2/EP1 C1-C128 扫描。recipe 包含各点的 prefill-decode
interval、prefill chunk、运行请求上限、显存与 SWA 前缀尾部设置，并追加对应的
changelog 条目。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(recipes): drop the sizing comment block from the GB200 DSV4.1-Flash SGLang recipe

Follow the AGENTS.md code comment rules: the block above the overrides
described the per-point configuration rather than explaining a specific
line. The parsed recipe and every bound per-point recipe are unchanged.

遵循 AGENTS.md 代码注释规范：删除 override 上方描述逐点配置的注释块。
解析后的配方以及每个测试点绑定的配方均保持不变。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Bryan Shan <58582368+Oseltamivir@users.noreply.github.com>

Enable native FP8 dispatch to benchmark NCCL EP low-latency decode in
FP8. Size receive buffers from the requested token ladder so larger
cases can run without the fixed LL token cap.

Co-authored-by: Bryan Shan <58582368+Oseltamivir@users.noreply.github.com>

…AgentX key / 新增 GLM-5.2 NVFP4 B300 Dynamo+SGLang 聚合 TP4 AgentX 配置 (#3953)

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>

…ectiveX：在 EFA 上运行 b300 EP16 并使用更快的 aws-ofi-nccl (#3872)

* CollectiveX: run b300 EP16 over EFA now that the nodes have gdrcopy

With gdrcopy installed the aws-ofi-nccl GIN plugin loads, but NCCL still
prefers its built-in GDAKI on the two ConnectX-7 IB ports, which are not the
GPU fabric, so window registration fails on most ranks and the rest hang.
The EFA profile now pins NCCL_GIN_TYPE=2 so GIN rides the Libfabric plugin.

DeepEP sizes its SMs from an ibstat probe that cannot see EFA devices; it
returns 0 and the model divides by it. A network.rdma_gbs registry field
(b300: 100 GB/s per GPU) is passed through instead.

Hand runs on two b300 nodes: nccl-ep and deepep-v2 EP16 bf16 decode and
prefill all pass the oracle.

现在 b300 节点已装 gdrcopy，CollectiveX 在 EFA 上运行 b300 EP16。装上 gdrcopy
后 aws-ofi-nccl 的 GIN 插件可以加载，但 NCCL 仍优先在两个并非 GPU 网络的
ConnectX-7 IB 端口上使用内置 GDAKI，导致多数 rank 窗口注册失败、其余 rank
挂起。EFA 配置现固定 NCCL_GIN_TYPE=2，使 GIN 走 Libfabric 插件。DeepEP 通过
ibstat 探测 RDMA 速率来确定 SM 数，但 ibstat 看不到 EFA 设备，返回 0 导致除零；
现由注册表字段 network.rdma_gbs（b300 每 GPU 100 GB/s）传入。两节点手动运行：
nccl-ep 与 deepep-v2 EP16 bf16 decode/prefill 均通过校验。

* CollectiveX: build aws-ofi-nccl 1.21.1 for multi-node EFA shards

GIN on EFA runs through the aws-ofi-nccl CPU proxy. The hosts mount 1.20,
whose proxy thread applies every GIN signal with an inline gdrcopy
read-modify-write of device memory; 1.21 hands signals to a worker thread
and drops the read. Multi-node EFA shards now build 1.21.1 from a
hash-pinned release tarball into the shared backend cache, against the host
libfabric, and the EFA profile loads it (net and tuner) in place of the
host's. On b300 EP16 bf16: deepep-v2 decode T=1 1825 -> 1011 us, nccl-ep
decode T=512 5145 -> 1576 us, nccl-ep prefill T=8192 72 -> 15.9 ms.

GPU-initiated GIN on EFA is still out of reach: it needs EFA installer
1.50.0 on the hosts (efa.ko 3.3.0 completion counters, libfabric 2.6) and
NCCL 2.31. The README now says so.

CollectiveX：为多节点 EFA 分片构建 aws-ofi-nccl 1.21.1。EFA 上的 GIN 经由
aws-ofi-nccl 的 CPU 代理。主机挂载的 1.20 版代理线程对每个 GIN 信号都内联执行
一次 gdrcopy 设备内存读-改-写；1.21 将信号交给工作线程并去掉读取。多节点 EFA
分片现从哈希固定的发布包针对主机 libfabric 构建 1.21.1 到共享后端缓存，EFA
配置用它（net 与 tuner）替代主机版本。b300 EP16 bf16：deepep-v2 decode T=1
1825 -> 1011 us，nccl-ep decode T=512 5145 -> 1576 us，nccl-ep prefill T=8192
72 -> 15.9 ms。EFA 上 GPU 发起的 GIN 仍不可用：需主机安装 EFA installer 1.50.0
（efa.ko 3.3.0 完成计数器、libfabric 2.6）及 NCCL 2.31，README 已注明。

* CollectiveX: bound the aws-ofi-nccl download so a stall retries

curl's --retry only fires once an attempt fails, and without a timeout a
stalled transfer never fails: backend preparation, and every peer node
waiting on the shared cache, would hang. Each attempt now gives up after
30 s to connect or 300 s in total, so --retry gets its chance.

CollectiveX：为 aws-ofi-nccl 下载设置超时，使卡住的下载能够重试

curl 的 --retry 只在一次尝试失败后才会触发，而没有超时设置时，卡住的传输
永远不会失败：后端准备步骤以及所有等待共享缓存的节点都会挂起。现在每次
尝试在连接超过 30 秒或总时长超过 300 秒后放弃，--retry 因此能够生效。

Align AgentX llm-d recipes with Sep 28 rank-level EPP routing and stop sweeping 2P×1D concurrency 576 after warmup stalls.

为 GB200 AgentX llm-d 启用每 DP rank 的外部负载均衡与 Rust 前端，并从主配置搜索空间中移除反复 warmup 失败的 2P×1D 并发 576。

…o-SGLang AgentX 配方 (#2978)

* feat(config): add Qwen3.5 B300 Dynamo-SGLang AgentX recipes

添加 Qwen3.5 NVFP4 B300 Dynamo-SGLang AgentX 配置和配方，并校验合成接受长度与已提交的黄金曲线一致。

* chore(changelog): link Qwen3.5 B300 PR #2978

在性能变更日志中补充 Qwen3.5 B300 PR #2978 链接。

* fix(config): extend Qwen3.5 AgentX Slurm limits

延长 Qwen3.5 AgentX 配方的 Slurm 时间限制。

* fix: remove temporary cache environment overrides

移除 Qwen3.5 B300 AgentX 配方中指向 /tmp 的缓存环境变量设置。

* feat(agentx): move Qwen3.5 B300 SGLang AgentX to nightly-dev-20261013-7224fce1

- Bump the four recipes and the master entry to
  lmsysorg/sglang:nightly-dev-20261013-7224fce1 and rename the recipe
  variants to match.
- Drop SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE so the NextN draft layer is served
  at its shipped precision.
- Replace aliases this image removed: cuda-graph-max-bs becomes
  cuda-graph-max-bs-decode (the old spelling is now an ambiguous prefix),
  and SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK=1 becomes
  SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0.
- Drop settings with no effect in this image: environment variables it no
  longer reads (SGLANG_ENABLE_SPEC_V2, SGLANG_ENABLE_FLASHINFER_GEMM,
  SGLANG_FLASHINFER_FP4_GEMM_BACKEND, SGLANG_CUTEDSL_MOE_NVFP4_DISPATCH),
  false booleans srt-slurm never passes (including the deprecated
  enable-dp-attention), and settings whose readers these TP-only aggregate
  workers never reach (DP-attention scheduler gathers, Mooncake and NVSHMEM
  variables, load-balance-method, moe-dense-tp-size, and
  hicache-io-backend: kernel, which page_first_direct rewrites to direct).
- Drop the recipe-level cpus-per-task directive, which conflicts with the
  b300-dsxe cpus-per-gpu default, and run the AgentX client from
  benchmarks/srt_agentic.sh.

将 Qwen3.5 B300 SGLang AgentX 迁移到 nightly-dev-20261013-7224fce1：

- 将四个配方与主配置条目升级到
  lmsysorg/sglang:nightly-dev-20261013-7224fce1，并相应重命名配方变体。
- 移除 SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE，使 NextN 草稿层以发布时的精度运行。
- 替换该镜像已移除的别名：cuda-graph-max-bs 改为 cuda-graph-max-bs-decode
  （旧写法现在是有歧义的前缀），SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK=1
  改为 SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0。
- 移除在该镜像中无效的设置：不再被读取的环境变量（SGLANG_ENABLE_SPEC_V2、
  SGLANG_ENABLE_FLASHINFER_GEMM、SGLANG_FLASHINFER_FP4_GEMM_BACKEND、
  SGLANG_CUTEDSL_MOE_NVFP4_DISPATCH），srt-slurm 从不传递的 false 布尔参数
  （包括已弃用的 enable-dp-attention），以及这些仅 TP 的聚合 worker 不会
  触达的设置（DP 注意力调度器 gather、Mooncake 与 NVSHMEM 变量、
  load-balance-method、moe-dense-tp-size，以及会被 page_first_direct 改写为
  direct 的 hicache-io-backend: kernel）。
- 移除与 b300-dsxe 默认 cpus-per-gpu 冲突的配方级 cpus-per-task 指令，并改为
  通过 benchmarks/srt_agentic.sh 运行 AgentX 客户端。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(agentx): use the default ReplaySSM cache length for Qwen3.5 B300 TP2/EP2

On lmsysorg/sglang:nightly-dev-20261013-7224fce1 the GDN ReplaySSM
spec-verify commit kernel (gdn_replayssm_compact_commit_kernel) contracts
over the replay window in tl.dot, which Triton requires to be at least 16.
With linear-replayssm-cache-len: 8 the TP2/EP2 worker fails to compile
that kernel at its first verify ("Input shapes should have M >= 1, N >= 1
and K >= 16"). Drop the override so the point uses the default length of
16, as the other recipes already do, and note it in the changelog entry.

在 lmsysorg/sglang:nightly-dev-20261013-7224fce1 中，GDN ReplaySSM 投机验证
提交 kernel（gdn_replayssm_compact_commit_kernel）在 tl.dot 中对回放窗口做
归约，而 Triton 要求该维度至少为 16。设置 linear-replayssm-cache-len: 8 时，
TP2/EP2 worker 在首次验证时无法编译该 kernel（"Input shapes should have
M >= 1, N >= 1 and K >= 16"）。移除该覆盖项，使该测试点与其他配方一样使用默认
长度 16，并在变更日志条目中加以说明。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* config: migrate qwen3.5 B300 agentic entry to srt-recipe-dir / srt-recipe

Main now requires srt-recipe-dir on srt-slurm configs (#3805). Move the
CONFIG_FILE additional-settings to row-level srt-recipe selectors; recipe
files and selectors are unchanged.

* chore: refresh PR #2978 for sweep reuse [skip-sweep]

* recipes: drop launcher-bound fields from qwen3.5 B300 agentic fragments

model.path/container/precision, identity.model.repo, identity.container.image
and RESULT_DIR/HF_HUB_CACHE/AIPERF_DATASET_MMAP_CACHE_DIR/IS_MULTINODE are
set by the launcher or workflow; main's fragment check rejects them (#3818).

* chore: refresh PR #2978 for sweep reuse [skip-sweep]

---------

Co-authored-by: Ankur-singh <ankusingh@nvidia.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…AgentX DSpark 配方 (#3738)

* feat(dsv4): refresh GB300 vLLM AgentX DSpark recipes with DEP8/DEP16 c384

Replace the GB300 vLLM AgentX MTP submission with nine DSpark K=6 configs
on DeepSeek-V4-Pro-0813, using 1p1d DEP8/DEP16 c384 in place of DEP4/DEP16 c256.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* perf-changelog: set PR link

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* perf-changelog: describe the GB300 vLLM AgentX DSpark submission as a whole

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(dsv4): bump GB300 vLLM AgentX Dynamo to 1.6.0.dev20261007

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(config): bump GB300 DSV4 vLLM AgentX Dynamo wheel to 1.6.0.dev20261010

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: refresh PR #3738 for sweep reuse [skip-sweep]

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Ankur-singh <ankusingh@nvidia.com>

# Conflicts:
#	inferencex-e2e/infx/tests/bench/test_fixed_seq.py
#	inferencex-e2e/perf-changelog.yaml