feat(trtllm): Qwen3.5 routed-expert block-FP8/MXFP8 refit

[hchings]

What does this PR do ?

Enable the TRT-LLM generation engine to run Qwen3.5 MoE with block-FP8 or MXFP8 routed experts, refit from the BF16 Megatron trainer at each step, plus the refit fixes and a GB200 nightly that go with it.

Issues

List issues that this PR closes (syntax):

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• ...

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[hchings]

Team review: Qwen3.5 routed-expert block-FP8 / MXFP8 refit

Reviewed by a team of 4 agents plus a leader re-verification pass over the full diff, with every upstream claim checked against the TensorRT-LLM commit this repo pins rather than main. 8 comments below.

The quantization module itself is good, and that's the bulk of the work here. Worth saying explicitly:

• quantization/fp8.py imports only math/re/torch and takes TRT-LLM types as parameters (configure_fp8_moe_backend(llm_kwargs, moe_config_type, ...)) rather than importing them. That's why the 223-line test file exercises the real quantization math on CPU with a 6-line stub and no engine mocking — a strictly better shape than the vllm/quantization/fp8.py analog, which routes the same decisions through a module-level mutable global.
• We checked the numerics against the pinned consumer and they're right: the VANILLA per-expert split is what Qwen3_5MoeHfWeightMapper expects, the [ceil(N/128), ceil(K/128)] FP32 scale orientation and dequant = fp8 * scale_inv match create_weights, the MXFP8 plain-uint8 [..., K/32] layout matches MXFP8CutlassFusedMoEMethod (which does its own packing and swizzle, so not pre-swizzling is correct), and all 13 modules_to_not_convert patterns translate correctly through upstream's Qwen3.5 normalizer. The block-FP8 caster was validated on CPU across aligned, unaligned, batched and non-contiguous inputs.
• The multi-worker result aggregation fix in trtllm_worker_async.py is a genuine improvement — the old results[0] masked failures on ranks > 0, and the updated tests cover it.

The findings share one root cause, which is why several read similarly: this was developed against a newer TensorRT-LLM than the repo pins, and the branch appears not to have been run against the pinned build. Three of the comments below are consequences of that (recompute_active_requests, the FP8 refit hooks, the torch.compile hooks); one is an unrelated rebase artifact that breaks all TRT-LLM runs (engine_cfg); and three are test/doc drift. None of them touch the design.

Generated by Claude Code

[hchings]

/ok to test b3f365f

[hchings]

/ok to test 6908199

[hchings]

/ok to test db997dd

[hchings]

/ok to test da64e72

[yuki-97]

Merge blockers — please rebase on main and resolve conflicts.

The PR delivers block-FP8 / MXFP8 refit for Qwen3.5 on TRT-LLM, and this round closes most of the earlier threads cleanly: the Dynamo packing rule, the always-BF16 set, the docs example and the stale comments are all fixed.

Scope: fix commits 2071d49 through da64e72, checked against the open threads from the previous review, plus a scan of the changed files for new issues.

Checked:

• Dynamo keeps the old packing rule end to end: use_legacy_packing goes from CollectiveSenderSpec through the synchronizer and lm_policy to both policy workers and the producer, and the legacy branch is byte-for-byte the old loop.
• Deciding linear weights from the built engine model works: the new real-model test (test_pattern_mode_quantizes_what_the_engine_holds_as_fp8) ran in the L0_Unit_Tests_Trtllm job at da64e72 and passed for dense block-FP8, dense MXFP8 and MoE block-FP8, with the router, shared_expert_gate and in_proj_* staying BF16.
• The MoE MXFP8 case of that test is skipped on SM90 and only the GB200 nightly covers it.

Remaining comments are on the routed-expert decision still going through pattern matching (in the thread on trtllm_backend.py), and on the supported-model wording and check (on refit.md).

[hchings]

/ok to test 591f1e0

[hchings]

/ok to test 6d36a23

[yuki-97]

Scope: fix commits e3da39a through 6d36a23, checked against the open threads from the previous review, plus a scan of the changed files for new issues.

Checked:

• Removing the pattern mode closes the open backend thread and the ignore-pattern threads: refit quantizes only names that fully match the routed-expert regex, and the fixed engine-form exclude list matches no expert module or ancestor at the pinned TRT-LLM f94eeb3738, so both sides agree for Qwen3.5 and Qwen3 MoE.
• validate_model_type runs on the same model_name TRT-LLM loads and resolves to qwen3_5_moe for the nightly's Qwen3.5-35B-A3B-Base.
• Both refit paths abort and re-raise on failure, BF16 and FP8 alike, so the engine-side cause reaches the caller.
• The MPI launcher hook only changes the environment in venvs that contain tensorrt_llm and keeps the root-allow variables the TRT-LLM worker sets.

Remaining comments are on the new job-wide Ray worker_process_setup_hook: it breaks runtime_env reuse in _init_ray on the cluster-tag-mismatch path, and the question of scoping it to the TRT-LLM generation path or dropping it is open to the author. The Qwen3 MoE "Verified" line on the open docs thread is also still pending, and the reply on the backend thread records that the pattern mode was dropped in favor of the fixed format.

nit, not blocking merging — a redundant local import importlib in virtual_cluster.py, and a leftover "By default" in refit.md.

[yuki-97]

for CI, will need to wait #4802 get into main first, otherwise CI will fail.

for #4122 (comment), seems all TRTLLM nightly will fail due to #4607, which is out of the scope of this PR.
@shuyixiong will fix it in another PR, thanks!

the newly added recipe in this PR validated pass locally w/ temporary fix for env vars, so kept it in this PR along.

[yuki-97]

/ok to test 9268b55