[None][feat] MiniMax-M3 NVFP4 low latency decode megakernel for Rubin (sm_107) TP2 / TP4 / TP8 in modeling_v2 by peihu-nv · Pull Request #20623 · NVIDIA/TensorRT-LLM

peihu-nv · 2026-10-19T19:16:44Z

Description

Depends on #20306 (flat staircase layout of the sm_103 targets). This PR adds only the last commit; please review that commit.

This PR brings the MiniMax-M3 NVFP4 decode targets to Rubin (sm_107): minimax_m3_nvfp4__sm_107__tp2, __tp4 and __tp8. They follow the flat layout of the sm_103 targets: one modeling.py per target, a weights.py weight table, and an ops/ catalog entry with certified cells and a test for every kernel the model calls. The router picks a target only for the matching checkpoint, SM and width, and modeling_v2: require makes it mandatory.

Performance

On Rubin (VR200), MiniMax-M3 NVFP4 AgentX with Eagle3, against the stock decode path (modeling_v2 off):

End to end: about 2.8× interactivity vs the best stock interactivity point (TP8 disaggregated, 1,112 tok/s/user, vs stock TP4 aggregated, 401).
Decode step: about 1.5× (TP2), 2.0× (TP4) and 2.4–2.7× (TP8) faster at batch 1.

sm_107 enablement

Catalog and routing: Arch.SM_107, and routing entries for (10, 7) at 2, 4 and 8 ranks, with the same drafter and runtime declines as sm_103.
Compiled kernels: minimaxM3Sm107Tp{2,4,8} with their trtllm::minimax_m3_sm107* thop ops, plus two shared GEN kernels: the TF32 tensor-core router GEMV and the spec-tail draft-input step.
CuTe DSL kernels: the decode MoE engines (1–8 rows, plus a 9–13-row engine at TP4), the MX GEMV, and the fused embedding + first RMSNorm.
Decode runtime: the step-input pack, skipped unchanged uploads, the spec step fold and the MSA decode metadata reuse. Each piece is inert until a target's RuntimeHooks turns it on, so other models are unchanged.
PrimTS (vendored FlashInfer task-scheduled attention): the dense layers and the drafter run its split-KV decode. The vendor patch scopes PrimTS's global WorkTileInfo rebind to its own compiles, so other CuTe DSL kernels (e.g. FA4) are unaffected.
KV fallback: the sm_107 NVFP4 → FP8 KV fallback skips MiniMax-M3 MSA, which keeps NVFP4 only on its sparse layers.
FMHA fallback: on SM107 the fallback FMHA serves paged prefill at the TRT-LLM Gen extra page sizes (the 128-token KV pages the MiniMax-M3 caches opt into) instead of refusing it; SM100 / SM103 keep the refusal.
Runtime items (target opt-in): the pinned FlashInfer CuTe DSL MXFP8 tactics above 32 rows on SM107, the post-warmup late-compile detector, and first-content emission (a request's first content after two iterations) are armed only when the model's target opts in through its RuntimeHooks; the MiniMax-M3 sm_107 targets do, and other models are unchanged. The AllReduce tuner leaves NCCL_SYMMETRIC out of its candidates only with TRTLLM_AR_TUNE_NCCL_SYMMETRIC=0 (opt-in; the default keeps the old candidates).

The targets

All three share one structure; they differ in rank geometry and row ranges.

	TP2	TP4	TP8
Decode step bound	192 rows (fast paths to 128)	48 rows	8 rows
MoE engine	1–8 rows (Triton finalize from 49)	1–8 + 9–13 rows	1–8 rows
Sparse decode attention	`m3_sparse_decode_lc`	`m3_sparse_decode_lc`	`m3_sparse_decode_lc8`

Boundaries: multicast all-reduce fused with the next RMSNorm and with the MoE finalize.
GEMMs: an MX GEMV for low-row MXFP8 projections and drafter GEMMs; at TP2 the LM head is split across the two dies.
Speculation (Eagle3, three drafts): a fused draft step and the target's own sampled accept. At TP4 the sampled verify runs two rounds (single-round accept off), matching the deployed configuration.
No tuning knobs: the targets read no tuning knobs from the environment; the only variable is the opt-in runtime-value dump diagnostic (TRTLLM_MODELING_V2_VALUE_DUMP), as in the sm_103 targets. Every fast path is on by default and has one module constant that turns it off. Capacity bounds carry a # capacity: annotation.

Aggregated (wide) decode at TP2

Rows: the TP2 target serves aggregated decode steps of up to 48 requests (1..192 rows, each request verifying 1 + 3 drafts). Fast-path cells cover up to 128 rows; 129..192-row steps take the stock-exact paths.
Wide cells: steps above 32 rows run the generic layer with 48-row multicast boundaries and MC finalize, the tensor-core router (9..48 rows), the Triton MoE finalize (49+ rows) and the 33..48-row GEMVs.
Mixed steps: steps with context rows run under breakable prefill CUDA graphs, with the drafter's attention in an eager break.

Numerics. No lossy steps. Most kernels are bitwise equal to the stock op they replace; the rest reduce in a fixed order, are deterministic in eager and graph mode, and stay within the stock kernel's error band. One known divergence: the TP4 drafter all-reduce + RMSNorm always sums like ONESHOT, so under a forced NCCL strategy its rounding differs from stock NCCL. Against the earlier wide TP2 reference, target tokens are identical: at 12+ concurrent requests under Eagle3 some draft tokens can resolve near-ties differently (target logits bitwise), and without speculation (not a production configuration) decode steps of 9..48 rows are bitwise on the first such step and differ after it, with identical output tokens. Steps of 1..8 rows are bitwise for prompts up to about 12K tokens, and at TP8 also for a single 83K-token request (prefill and decode).

Not in this PR: batch 4/8 decode at TP4 / TP8 (follow-up). The prefill (context) optimizations are in #20629.

Known behaviours: The pinned SM107 MXFP8 GEMM tactics are armed process-wide by an executor whose model opts in, so another model in the same process at the same GEMM shape also uses them; its outputs are bitwise unchanged (every pinned tactic equals the default), and the LLM API runs one executor per worker process.

Test Coverage

Bitwise A/B vs the pre-restructure sm_107 targets (TP2/TP4/TP8, no speculation and Eagle3, every rank, prompts up to about 12K tokens; at TP8 also one 83K-token request): identical tokens and per-step output hashes. The only exception is the unread tail of new_tokens in Eagle3 context steps. Re-run on the final commit at TP2 and TP4 (no speculation and Eagle3): bitwise at every step of 1..8 rows; at TP2 the first difference is a 45-row context step, which this PR moves to the wide cells.
Wide TP2 vs the aggregated (wide) reference target at 9..128 rows, Eagle3 and no speculation: target tokens identical in every phase and logits bitwise on every recorded step under Eagle3; the two deltas above are the only differences.
FMHA fallback at 128-token pages on SM107: a GPU oracle (fresh contexts beside generation rows, cached prefixes intact, K/V stored at their slots) and a CPU gate, all cases in band.
Unit tests on Rubin, one session per file: TP2 1449, TP4 1659, TP8 1415 tests, all passing. Each target's drafter context cell is a strict expected failure because the SM107 non-paged context FMHA does not apply RoPE (shared runtime issue, tracked separately).
Decode runtime tests (executor/, speculative/, attention): 255 pass, 1 skipped.
Delivery and capacity checkers: TP2 99/99, TP4 107/107, TP8 97/97.
Accuracy at TP2 vs the pre-restructure targets: GSM8K (1319) McNemar p = 0.30 greedy, p = 1.00 sampled; the Eagle3 natural acceptance length over 264 long-context requests is identical. At TP4 and TP8, GSM8K (1319) vs the pre-restructure targets: TP4 p = 1.00 greedy, p = 1.00 sampled; TP8 p = 0.38 greedy, p = 0.29 sampled.
Decode perf vs the pre-restructure targets, same node, both run orders: TP2, TP4 and TP8 within noise (TP8 compared at equal decode work: the synthetic runs' generated text, and so the number of distinct routed experts per step, varies from server start to server start).

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

… step-loop runtime hooks Adds the modeling_v2 target minimax_m3_nvfp4__sm_103__tp4 (GB300, TP4, Eagle3 one-model speculation) and the executor seams it opts into. - Runtime hooks (off unless a model declares them): packed step inputs, skipped unchanged H2D uploads, a private graph metadata pool, sampler greedy-token and step-fold paths, MSA runtime policy, and rank-merged autotuned tactics. Greedy-only outputs are kept only when nothing reads the logits (no guided decoder, non-empty batch). - TP4 decode target: fused attention with long-context selection, a pre-wait-staged NVFP4 QKV/indexer producer and index scorer, multicast all-reduce + RMSNorm boundaries with the MoE finalize folded in, MXFP8 GEMVs, CuTe DSL MoE engines, an exact greedy and sampled verify, a fused draft loop, and lossless 12-bit coded BF16 target and draft heads. - Explicit dispatch: every covered shape is a certified catalog cell; under modeling_v2: require a covered shape that cannot run raises. Every item is on by default with one constant that turns it off; no environment reads. - Numerics match stock up to summation order; the long-context selector breaks >1024-way score ties by its contract (smaller block). Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

Pick the row source explicitly, raise when neither logits nor greedy tokens are present, and assert logits before the fast greedy kernel. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

Adds the modeling_v2 target minimax_m3_nvfp4__sm_103__tp8 (GB300, 8 ranks over two trays, tensor-parallel attention and experts, one-model Eagle3 or no speculation) and routes world = tp = moe_tp = 8 to it; the tp4 branch of the router is unchanged. - Decode path: the tp4 target's exact decode path at the TP8 geometry, plus the TP8 items: the 1..8-token MoE engine at expert width 384 (logits replicated 32x at <= 4 tokens), an 8-head one-launch long-context sparse decode staged before its grid wait at <= 8 rows, a fused index score + top-16 select for spans of <= 2 requests, the producer releasing its dependents on entry, the long-row select's early release at <= 8 rows, the TP8 MXFP8 GEMV table, fixed-tactic low-M BF16 GEMMs for the drafter linears, the drafter decode's KV split capped at 16K for one request, BF16 target and draft heads, and the drafter fc forked beside the verify head. - Isolation: the target imports nothing from the tp4 target; the MoE engine and the MX GEMV op it changes are its own copies inside the target. The two new kernels are compiled ops (cpp/tensorrt_llm/kernels/minimaxM3Tp8, trtllm::minimax_m3_tp8_*). A drift test pins the files copied unchanged from the tp4 target. - fmha/flashinfer_trtllm_gen.py applies a layer's KV-split cap (msa_decode.dense_decode_max_seq_len); a no-op unless a layer sets one. - Same contract as the tp4 target: every kernel a catalog entry with CELLS and a torch reference, dispatch.py listing each fast path's rows, a covered shape that cannot run raising under modeling_v2: require, kernels built before warmup, every item on by default with one constant to turn it off. - Numerics match stock up to summation order (BF16 heads and drafter, NVFP4 experts, MXFP8 projections). Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

CUtensorMap already carries its own alignment, which is stricter than 64 bytes in newer CUDA toolkits; clang-hosted nvcc rejects the weaker alignas with "alignment cannot be set to less than the default alignment" (SBSA build). Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

Resolve two additive conflicts with the Qwen4-exp modeling_v2 target by keeping both router entries and both routing tests. Port the sampler's greedy_tokens_usable to main's request_row_params after the TorchSampler strategy refactor (NVIDIA#20188) removed _request_strategy. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

- Register fake implementations for the new step_inputs_scatter and MiniMax-M3 router, low-M GEMM, producer and allreduce-norm ops. - Give the warmup tests' stand-in engines the default runtime hooks the decoder runner now reads. - Leave the identity table unset in the piecewise-attention metadata mock. - Stub the eager K/V view resolution only while the runtime-policy fixture builds its manager, so the pool-view tests still exercise get_buffers. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

Restructure the minimax_m3_nvfp4__sm_103__tp4 and __tp8 targets into the flat layout the modeling_v2 exemplars use, with no change to what they compute. Each target owns its own copy of every file (the targets share no code): * modeling.py: the dispatch tables (FAST_PATHS, row ranges, miss/uncovered policy), the flat core, PrefillTarget / DecodeTarget, the one-model Eagle3 drafter and worker, the step fold, runtime hooks and the shell. The core selects the target on num_contexts alone: every pure generation step is DecodeTarget's (its decode kernels up to 48 tokens, its own generic layers for wider steps or embeddings), every step with context rows PrefillTarget's. The two targets share no body: each owns its layer, attention and MoE compositions, and its step_args is the one projection of the MSA metadata its forward reads (DecodeTarget states num_contexts=0 / num_ctx_tokens=0). Shared module-level code is limited to single-call op adapters that read no step metadata. * weights.py: the language-model and drafter weight tables (host-side TP slicing, consumed checkpoint keys released during the load; at TP8 the 4 KV / index heads are duplicated on rank pairs as the built-in loader does). * ops/: catalog entries with certified cells for every call the model makes; kernel wrappers route through their entries. The MSA and drafter attention entries carry a numeric reference and GPU tests that drive every cell against real caches and prepared metadata. * kernels/: the CuTe DSL GEMV, MoE engine and embedding+first-norm kernels move into each target (the old shared cute_dsl_kernels/minimax_m3 copy is removed); capacity annotations on every constant. * The legacy dispatch/layers/linears/runtime/spec_fold/spec_worker modules are removed; the unit tests are ported to the flat targets. * The unfused QKV / index projection runtime (fuse_qkv_index_projection=False, FP8 indexer, FP8 or NVFP4 main KV) runs as the built-in model does. Routing (models/minimax_m3/routing.py) now declines what the targets do not build or run, so modeling_v2="auto" falls back to the built-in model and "require" names the missed criterion: * Eagle3 drafters other than the targets' one-layer drafter (read from the drafter checkpoint's config.json). * Runtimes other than MSA with an FP8 indexer and an FP8 or NVFP4 main KV cache. The resolver hands routing modules that set ROUTES_ON_RUNTIME a RuntimeFacts (sparse attention config and the KV cache quantization the engine will run with); only MiniMax-M3 sets it. _validate_runtime stays as the backstop. Adds a whole-model accuracy test under modeling_v2="require" (GSM8K without speculation, and with one-model Eagle3 plus a chat-GSM8K acceptance probe), on the 4-GPU GB300 list (TP4) and the 2-node GB300 list (TP8). Bitwise identical to the previous targets on GB300 (tokens and per-step outputs on every rank, no speculation and Eagle3, fused and unfused projections, TP4 and TP8, including pure generation steps wider than the decode kernels' 48 tokens). Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

Resolve cache_manager.py: keep the decode shortcuts and pooled views, and the HND main KV storage with the strided NHD view from NVIDIA#20159. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

peihu-nv · 2026-10-19T19:34:03Z

/bot run --disable-fail-fast

…ls in M3 folders Review follow-up, no logic change: - cpp/tensorrt_llm/kernels/stepInputsScatter.{cu,h} move to kernels/minimaxM3/. - The MiniMax-M3 torch ops (thop/minimaxM3*Op.cpp and stepInputsScatterOp.cpp) move to thop/minimaxM3/, which adds them to th_common_attention with target_sources() (as thop/moe/ does for th_common_moe). - The fake impls of the ops the decode targets add (step_inputs_scatter, minimax_m3_mxfp8_low_m_gemm, minimax_m3_router_gemv / _gemv_fp4 / _tc, minimax_m3_producer_pw, minimax_m3_mc_allreduce_norm) move from custom_ops/cpp_custom_ops.py to modeling_v2/models/minimax_m3/fake_ops.py, imported by the MiniMax-M3 modeling_v2 package, by pyexecutor/step_inputs.py and by the fake-registration test. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

…ough runtime_hooks_of The autotuner and CUDA-graph warmup passes read merge_tuned_tactics_across_ranks from the runner's cached hooks, which only the constructor sets; engines built another way (main's warmup tests drive DecoderRunner._run_autotuner_warmup on a stub) failed with an AttributeError. They now read runtime_hooks_of(self.model), the all-off default for a model that declares no hooks, the same value the constructor caches. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

… HND storage layout Since NVIDIA#20159 the dense KV storage is head-major and get_buffers hands a token-major request a transposed view. The reused pool views (reuse_pool_views) cached the untransposed view and transposed it on every call, so a reused view was a new tensor each time. The transpose is now part of the view's cache key and applied before caching. The runtime-policy tests expect the view in the manager's own layout over its physical one. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

Resolve runner.py after NVIDIA#20134 split DecoderRunner by phase: the model's runtime hooks now reach the phase that uses them. The runner keeps the step scopes, the graph-runner config and the greedy-token logits; the input preparer the H2D skip copies, the step-input pack and the speculative step fold; the metadata the MSA policy and the greedy RNG upload skip; the graphs the short/long split and the post-capture tactic merge; the warmup the post-autotune tactic merge (through runtime_hooks_of). Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

peihu-nv · 2026-10-20T02:27:42Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-20T02:35:50Z

PR_Github #79103 [ run ] triggered by Bot. Commit: 5d24e81 Link to invocation

…ares is_spec_decode DecoderRunner reads DecoderRunnerConfig.is_spec_decode at construction (the spec step fold binds the engine buffers). The compiled-body validation test builds the runner from a SimpleNamespace config; give it the field the real config always has. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

Resolve the modeling_v2 router against NVIDIA#20267: route() now takes the resolver's KV-cache algorithm as its fifth argument for every routing module, and modules that set ROUTES_ON_RUNTIME (MiniMax-M3) still receive runtime=RuntimeFacts; the MiniMax-M3 route() accepts the new argument. Keep both sides' new helpers (RuntimeFacts / resolved_mode and tp_reaches_across_nodes) and both test-list additions. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

tensorrt-cicd · 2026-10-20T07:39:28Z

PR_Github #79103 [ run ] completed with state FAILURE. Commit: 5d24e81
/LLM/main/L0_MergeRequest_PR pipeline #65162 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

peihu-nv · 2026-10-20T12:18:03Z

/bot run --disable-fail-fast

Resolve the MiniMax-M3 index-score kernels against NVIDIA#19275 (c146094), which split cute_dsl_kernels/ into the domain kernel trees: the TP4 and TP8 m3_index_score imports now take cute_ptx_utils and utils from tensorrt_llm._torch.kernels.blackwell, and the docstring points at attention/kernels/blackwell/minimax_m3_index_decode_score.py. Both moved modules are unchanged (pure renames). Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

… (sm_107) TP2 / TP4 / TP8 in modeling_v2 Three modeling_v2 targets for the MiniMax-M3 NVFP4 checkpoint on Rubin (sm_107), minimax_m3_nvfp4__sm_107__tp2, __tp4 and __tp8, in the flat layout of the sm_103 targets: one modeling.py per target, a weights.py weight table and an ops/ catalog entry with certified cells and a test for every kernel the model calls. Routing picks a target only for the matching checkpoint, SM and width. sm_107 enablement: Arch.SM_107 in the catalog and the (10, 7) routing entries; the compiled trtllm::minimax_m3_sm107* kernels (built for sm_107 only; torch ops in thop/minimaxM3/, fake impls in the M3 fake_ops module) and the shared TF32 router GEMV and spec-tail draft-input kernels; the CuTe DSL decode MoE engines, MX GEMV and fused embedding + first RMSNorm; the decode runtime pieces (step-input pack, graph runner hooks, step fold, MSA decode metadata reuse, the SM107 NVFP4 -> FP8 KV fallback skipping MiniMax-M3 MSA, late-compile detection), each inert until a target's runtime hooks turn it on; the FMHA fallback serving SM107 paged prefill at the TRT-LLM Gen extra page sizes (the 128-token KV pages its MiniMax-M3 caches opt into); and the scoped PrimTS work-tile rebind in the vendored FlashInfer patch. The TP2 target serves decode steps of 1..192 rows: context steps of 9..48 rows route on the tensor-core router, steps with context rows run under breakable prefill CUDA graphs (the drafter's attention in an eager break) and take the MC finalize boundary at 9..48 rows, and decode steps of >= 33 requests take the stock MSA merge path (MergeMxfp8.MAX_ROWS 128; no canon profile exceeds 96 rows). Runtime items: pinned FlashInfer CuTe DSL MXFP8 tactics above 32 rows on SM107, for models whose runtime hooks opt in (the MiniMax-M3 sm_107 targets do); the AllReduce tuner leaves NCCL_SYMMETRIC out of its candidates only when TRTLLM_AR_TUNE_NCCL_SYMMETRIC=0 (opt-in; the default 1 keeps the old candidates); PyExecutor emits a request's first content after two iterations when the model's runtime hooks ask for it (the MiniMax-M3 sm_107 targets do; other models keep the stock schedule). No lossy steps: every kernel keeps the stock op's precision. The TP4 and TP8 targets produce bitwise the same tokens and per-step outputs as the pre-restructure sm_107 targets, with and without Eagle3. Two known deltas against the previous wide TP2 target (the 9..128-row cells this commit folds in), both measured with greedy sampling: under Eagle3, at 12 or more concurrent requests the drafter's BF16 linears past its low-M tactic table can resolve a near-tie differently from the in-tree drafter, so some draft tokens differ, while the target's tokens are identical and its logits bitwise; without speculation (not a production configuration), decode steps of 9..48 rows are bitwise on the first such step and differ after it (cause not isolated), with identical output tokens in every comparison. Steps of 1..8 rows are bitwise to the previous TP2 target. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

peihu-nv · 2026-10-20T15:22:21Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-20T15:31:17Z

PR_Github #79291 [ run ] triggered by Bot. Commit: 7333ac8 Link to invocation

tensorrt-cicd · 2026-10-20T20:21:49Z

PR_Github #79291 [ run ] completed with state SUCCESS. Commit: 7333ac8
/LLM/main/L0_MergeRequest_PR pipeline #65314 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yuxianq · 2026-10-22T00:00:11Z

@farazkh80, could you help to review this PR as trt-llm-torch-attention-devs? Thanks~

Assignment explanation

Components	kernels/msa (45%); kernels/auxiliary (23%); kernels/prims-ts (8%); kernels/trtllm-gen (6%); runtime/sparse-cache (5%); fmha/fallback (5%); fmha/msa-sparse-gqa (4%); fmha/flashinfer-trtllm-gen (2%); integration/build-config (1%); integration/dependencies (1%)
Tags	algorithm:sparse (75%); algorithm:dense (25%); specific-algorithm:minimax-m3 (100%); representation:gqa (100%); phase:decode (90%); phase:prefill (10%); features:quantization (40%); features:rope (20%); features:qk-norm (15%); features:paged-kv (15%); features:speculative-decoding (10%); context:llm (100%)
PR effort	13 points — The visible contracts require deep review of sparse reductions, TP geometry, quantized preprocessing, copy ordering and shared attention dispatch/scheduling. Their combined complexity supports 13 attention-review points independently of file count or renames. This extends execution for MiniMax-M3; no new model architecture is established.
Importance	Important — sparseDecodeLc2.cu describes clustered sparse attention with cross-CTA merging and quantized operands; denseQkNormRopeFp8.cu specifies sensitive two-stage rounding. fallback.py also adds an SM107-specific paged-context exception. These expose substantial correctness and performance contracts.
Retained GitHub review requests	litaotju: preserve direct code-owner review request; schetlur-nv: preserve other-group coverage: NVIDIA/trt-llm-disagg-devs, NVIDIA/trt-llm-kv-cache-manager-devs, NVIDIA/trt-llm-oss-compliance, NVIDIA/trt-llm-models-devs, NVIDIA/trt-llm-kernels-devs, NVIDIA/trt-llm-devs, NVIDIA/trt-llm-runtime-devs, NVIDIA/trt-llm-multi-gpu-ci-review
Affinity (0–1)	0.593 component/tag match; 0.445 after PIC preference
Outstanding points at selection	25 → 38 (+13)
Points already reserved on this PR	0
Effective weight	1
Completed review points (preceding 168 hours)	58
Combined points after assignment	96
Demand-derived target	76.00 points (soft share)
Additional fair share before assignment	0.00 points
Share explanation	Whole PR / qualification or coverage constraint exceeds a soft share; assignment is allowed.
Selection rule	affinity with 0.3-share guard
Assignment scope (farazkh80)	all

Affinity and workload are snapshots at selection, not live counters. Existing reservations on this PR are counted once.

…efault device The import-time probe built its tensors without a device. When the module is first imported while a model is built under the meta device, the probe tensors are meta and torch.equal raises. Pin them to the CPU, as the comment already states. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

peihu-nv · 2026-10-22T01:19:51Z

/bot run --disable-fail-fast

litaotju · 2026-10-22T03:21:27Z

@@ -0,0 +1,3 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+"""MiniMax-M3 NVFP4 / sm_103 / tp4."""


@peihu-nv , one thing I want to clarify is that, we dont require ONE modeling folder to support just ONE EXACT parallel config, it can be TP general if most of the code are shared between different TP sizes.
See this example https://github.com/NVIDIA/TensorRT-LLM/pull/20267/changes

There is no strict threshold on what percentages of differences leads to specialization, you can use your best judgement.
But my normal mental model is that TP 2/4/8 may not be that different in terms of kernels/fusion patterns?

You can decide, if this TP2/4/8 specialization is what you intend, its also fine, just leave a msg.

tensorrt-cicd · 2026-10-22T05:46:36Z

PR_Github #79695 [ run ] completed with state FAILURE. Commit: ffa76ab
/LLM/main/L0_MergeRequest_PR pipeline #65615 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

mzweilz · 2026-10-22T09:24:10Z

+        self._gsm8k(eagle3=True)
+
+
+class TestModelingV2MiniMaxM3Nvfp4Sm107Tp2(_MiniMaxM3Target):


These SM107 accuracy cases are not registered in a matching CI stage; the added GPU entries exercise SM103.

farazkh80

Approved from attention perspective, with a couple questions (non-blocking).

farazkh80 · 2026-10-22T21:25:14Z

+        {
+            return;
+        }
+        if (isContextKernel(runnerParams.mKernelType))


The new context warmup grid runs on every arch, but on SM100/SM103 no context kernel goes through NVRTC, so it only adds startup time there. Should we gate it on SM107?

farazkh80 · 2026-10-22T21:25:14Z

+    # MiniMax-M3 MSA keeps NVFP4 only on its sparse layers, which MSA and
+    # Triton serve; its dense and draft layers stay FP8 for trtllm-gen.
+    sparse_config = model_config.sparse_attention_config
+    uses_minimax_m3_msa = (isinstance(sparse_config,


With modeling_v2 off by default, this also moves stock MiniMax-M3 on Rubin to NVFP4 sparse KV. Was that path checked on Rubin, or only the sm_107 targets?

peihu-nv added 8 commits October 14, 2026 20:10

[None][fix] Satisfy mypy on the MiniMax-M3 sampler fast-greedy path

c76faa4

Pick the row source explicitly, raise when neither logits nor greedy tokens are present, and assert logits before the fast greedy kernel. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

[None][chore] Merge main into MiniMax-M3 megakernel PR

6d9af78

Resolve cache_manager.py: keep the decode shortcuts and pooled views, and the HND main KV storage with the strided NHD view from NVIDIA#20159. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

github-actions Bot assigned peihu-nv Oct 19, 2026

peihu-nv changed the title ~~[None][feat] MiniMax-M3 NVFP4 decode targets for Rubin (sm_107) TP2 / TP4 / TP8 in modeling_v2~~ [None][feat] MiniMax-M3 NVFP4 low latency decode megakernel for Rubin (sm_107) TP2 / TP4 / TP8 in modeling_v2 Oct 19, 2026

peihu-nv added 3 commits October 19, 2026 12:40

peihu-nv mentioned this pull request Oct 19, 2026

[None][perf] Optimize MiniMax-M3 NVFP4 context (prefill) path on Rubin (SM107) #20629

Draft

1 task done

peihu-nv force-pushed the peihengh/m3-vr-sm107-decode branch from cd5f671 to 5d24e81 Compare October 20, 2026 02:27

peihu-nv added 2 commits October 19, 2026 20:18

peihu-nv force-pushed the peihengh/m3-vr-sm107-decode branch from 5d24e81 to 4d16da2 Compare October 20, 2026 12:17

peihu-nv added 2 commits October 20, 2026 05:40

peihu-nv force-pushed the peihengh/m3-vr-sm107-decode branch from 4d16da2 to 7333ac8 Compare October 20, 2026 15:21

peihu-nv requested review from a team, Wanli-Jiang, WeiHaocheng and litaotju as code owners October 21, 2026 23:49

peihu-nv requested review from VALLIS-NERIA, bo-nv, cascade812, chuangz0, heyuhhh, joyang-nv, lowsfer, mzweilz, niukuo and schetlur-nv October 21, 2026 23:49

yuxianq requested review from farazkh80 and removed request for heyuhhh October 22, 2026 00:00

litaotju reviewed Oct 22, 2026

View reviewed changes

mzweilz reviewed Oct 22, 2026

View reviewed changes

farazkh80 approved these changes Oct 22, 2026

View reviewed changes

Wanli-Jiang mentioned this pull request Oct 23, 2026

[None][feat] modeling_v2: build target C++/CUDA as per-family libraries #20896

Draft

1 task done

		self._gsm8k(eagle3=True)


		class TestModelingV2MiniMaxM3Nvfp4Sm107Tp2(_MiniMaxM3Target):

Conversation

peihu-nv commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Performance

sm_107 enablement

The targets

Aggregated (wide) decode at TP2

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

peihu-nv commented Oct 19, 2026

Uh oh!

peihu-nv commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

peihu-nv commented Oct 20, 2026

Uh oh!

peihu-nv commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

yuxianq commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peihu-nv commented Oct 22, 2026

Uh oh!

litaotju Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

tensorrt-cicd commented Oct 22, 2026

Uh oh!

mzweilz Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

farazkh80 left a comment

Choose a reason for hiding this comment

Uh oh!

farazkh80 Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

farazkh80 Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

peihu-nv commented Oct 19, 2026 •

edited

Loading

yuxianq commented Oct 22, 2026 •

edited

Loading