[Roadmap] Quantization 2026 H2

[OrangeRedeng]

SGLang Quantization Roadmap

Planning horizon: 2026 H2
Last updated: 2026-07-29

This roadmap consolidates and updates:

• NPU Quantization Roadmap #14424 — closed
• Quantization Modifications Roadmap #15194 — open
• Ascend A5 MXFP8/MXFP4 RFC #21584 — open
• FP4 KV Roadmap #29913 — open

Status policy:

• [x] means the linked PR was merged or the tracking issue explicitly reports completion.
• [ ] means work is open, incomplete, unowned, or still needs production validation.
• A closed PR without merge is not considered complete; replacements are identified where known.

Checklist

• If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
• Please use English. Otherwise, it will be closed.

Motivation

Quantization support has expanded beyond weight-only linear layers into MoE, attention, KV cache, communication, VLM, diffusion, speculative decoding, and disaggregated serving. However, checkpoint loading, parameter creation, scheme selection, backend dispatch, post-load processing, and kernel execution are still coupled in several paths.

The target architecture is:

Config → Method → Scheme → Kernel

The same kernel should be reusable by multiple checkpoint formats, while checkpoint parsing and weight creation remain format-specific. Offline checkpoint loading and online runtime quantization must also have separate, predictable public interfaces.

1. Architecture, Dispatch, and Weight Lifecycle

1.1 Scheme and kernel separation

• Compressed-Tensors MoE schemes: #17503
• ModelSlim MoE schemes: #17993
• Quark MoE schemes: #18252
• AWQ schemes and kernel/weight-init split: #21126
• GPTQ schemes and kernel split: #26402
• CPU GPTQ scheme refactor: #26786
• ModelOpt scheme refactor: #20963 @wenscarl
• AutoRound package/scheme split: #26846 @LProDreamAll
• GGUF hardware-backend and scheme split: #26942 @LProDreamAll
• Ascend MoE refactor and removal of the legacy forward_npu path: #25663, #25678
• Delegate the Ascend MXFP8 dense scheme to the kernel abstraction: #28505
• Complete the unified sglang.kernels migration: RFC #29630, #32072, #32160
• Migrate Compressed-Tensors W4A4 NVFP4 MoE to MoeRunner: #32248
• Remove unused kernel implementations and stale registry entries: #32636
• Migrate the SM100 TRT-LLM-generation MXFP4 MoE path to MoeRunner: #32405 @Beihao-Zhou

Closed without merge:

• #17361 — original Torch-NPU MoE runner refactor; superseded by #25663 and #25678

1.2 Weight loading, post-load processing, and online quantization

• Complete Weight Loader v2 #31051, including dense, MoE, MLA, hybrid, multimodal, and wrapper migrations. The initial demo #28671 is merged, but the PR1–PR9 migration remains incomplete.
• Separate tensor loading from checkpoint-completeness verification: #28585
• Standardize process_weights_after_loading across ModelOpt, Compressed-Tensors, ModelSlim, Quark, AutoRound, AWQ, GPTQ, GGUF, speculative layers, and weight-update paths.
• Use --quantization only for pre-quantized checkpoints and --online-quantization only for runtime quantization.
  • MXFP8 online-quantization foundation: #17449
  • ROCm BF16/FP8 to MXFP4 online paths: #18005, #18182, #28291
  • FP8 online-quantization fix: #26415
  • Ascend W8A8 INT8 online quantization for MoE: #29048 @OrangeRedeng
  • ROCm NVFP4-to-MXFP4 online requantization: #29328 @ColinZ22
  • Fix the MXFP8 online-quantization pipeline: #31510
  • Add the same explicit runtime-quantization interface to diffusion. #23373 was closed without merge.
• Preserve registered NPU parameters and loader metadata during RL weight updates: #31796
• Support native heterogeneous weight conversion across TP/PP/EP/DP layouts: #32335
• Support NVFP4 in the model weight-loading daemon: #32398 @xuantengh
• Propagate each model's packed_modules_mapping into its quantization configuration: #32739 @Hert4

1.3 Capability registry and validation

• Introduce machine-readable compatibility metadata for format, scheme, backend, platform, layer type, scale granularity, kernel, and parallelism: #31774
• Restore serialized Marlin checkpoint detection, registry resolution, and native kernel dispatch: #32250, #32644 @gitover22
• Prevent mixed-precision Compressed-Tensors checkpoints from silently loading unquantized: #32736 @Hert4
• Report only platform-supported quantization methods in validation errors: #32253 @MrAnayDongre
• Fail early for unsupported FP4/backend combinations: #32333 @zooeyQAQ
• Add Ascend CI coverage for alternate checkpoint prefixes such as w1/w2/w3 and gate/up/down_proj: #31570
• Generate the public support matrix, documentation, and recurring CI matrix from capability metadata.
• Make DSA backend defaulting and validation table-driven as the first capability-registry slice: #32576 @mosya415

2. Linear and MoE Quantization

2.1 NVIDIA CUDA

• Support ModelOpt NVFP4_AWQ checkpoints: #31825
• Support ModelOpt MXFP8 checkpoints, scale-name conversion, and FP8 KV-cache detection: #32538 @mmangkad
• Add SM90 MXFP8 inference for MiniMax-M3 with DeepGEMM: #31791 @DaZhUUU
• Support nvidia/MiniMax-M3-NVFP4: #31989 @trevor-m
• Switch the DeepSeek-R1 output projection GEMM to MXFP4: #31775 @heachary
• Serve ModelOpt mixed-precision checkpoints with block-FP8 attention and shared experts: #31855 @Fridah-nv
• Support static W4A8 with FP8 activation quantization for MoE: #32173 @ZeroKernel798
• Land the Humming W1–W8 kernel foundation: #23754
• Add native Humming W4AFP8 checkpoint schemas: #32033 @guzekai01
• Complete the NVFP4 improvement track: #26543 @THUCBQ
• Reduce MXFP4 Marlin repack memory: #31898 @zzt93
• Use cuBLAS for per-tensor FP8 GEMM on SM120: #31961
• Select FlashInfer CUTLASS for automatic NVFP4 MoE on SM100: #32229 @janbernloehr
• Reduce reserved GPU memory during UE8M0 group requantization: #31755 @CSWYF3634076
• Apply gemm1_clamp_limit in SiLU/mul/quant kernels without gemm1_alpha: #31935 @ChiachiHsieh
• Zero-initialize masked-path UE8M0 activation scales: #32385, #32386 @yueming-yuan
• Fix H100 DeepEP TBO scale lifetime and non-finite quantization handling: #32188
• Restore numerically stable FP8 per-token group quantization on Hopper: #32616
• Fuse Kimi-K3 routing, TRT-LLM packing, and MXFP8 activation quantization: #32699
• Make RowParallelLinear K-size handling tuple-aware for FP8: #30742
• Fix the NVFP4 CUDA-graph nightly crash: #32118
• Optimize the GLM-5.2 FP8 low-latency serving stack on multi-node GB300: #32633 @BBuf
• Use parameterized SiTU for the Kimi-K3 FlashInfer MXFP4 path: #32650 @zhangxiaolei123456
• Enable FlashInfer MXFP4 MoE for GPT-OSS on SM120: #32668 @mmangkad
• Fix Nemotron Nano mixed MXFP8/NVFP4 loading and empty output: #32740 @danielafrimi
• Consolidate or remove legacy GPTQ/AWQ/attention kernels only after coverage is preserved: #32112, #32114
• Fuse static FP8 activation quantization into producer kernels: #31504
  • RMSNorm + FP8 quantization: #31963 @mattteochen
  • SiLU/clamp/mul + FP8 quantization for EP MoE: #32058 @HX-1234
  • Gated RMSNorm + FP8 quantization for Qwen3.5: #32443 @JustinTong0323

Closed without merge:

• #32431 — SM80/SM86 fallback and AWQ MoE loader support; replacement TBD

2.2 AMD ROCm

• Keep AITER imports optional on installations without AITER: #31805 @nemanjaudovic
• Online MXFP4 from BF16 and FP8: #18005, #18182, #28291
• Complete NVFP4-to-MXFP4 online requantization: #29328 @ColinZ22
• Support Quark PTPC FP8 attention on MLA models: #28734 @ColinZ22
• Support Quark DeepSeek-V4 MXFP4 with shared experts and MTP: #31054 @ColinZ22
• Enable MiniMax-M3 MXFP8 MoE:
  • Unblock MXFP8 block conversion on gfx950: #32036 @yctseng0211
  • AITER MoE: #32133 @YukioZzz
  • AITER FlyDSL native MXFP8 SwiGLU: #32190 @yctseng0211
• Add GLM-5.2 MXFP4 disaggregated-serving CI recipes: #32120 @Lzy17
• Route gfx950 A8W8 quantization to AITER: #32506 @Fyzyukk
• Enable diffusion BF16 attention and MXFP4 compilation on ROCm: #27702 @avjves
• Enable opt-in AITER CK W8A8 block GEMM on gfx942: #32516 @ThomasNing
• Add recurring GLM-5.2 FP8 accuracy CI on MI35x: #32570 @michaelzhang-ai
• Add gfx1250 FP8/MXFP4 dense, MoE, attention, and communication coverage: #32754 @akao-amd

2.3 Ascend NPU

• W4A4 dense linear: #14504
• W4A8 activation clipping: #14736
• W4A4 MoE: #18924
• Checkpoint frameworks:
  • AWQ: #10158
  • Compressed-Tensors: #14504
  • GPTQ dense and MoE: #15203, #16364
  • AutoRound: #16699
  • GGUF dense and MoE: #17883
• Qwen3 dense MXFP matrix:
  • W8A8 MXFP8: #22352
  • W4A8 MXFP4: #23650
  • W4A4 MXFP4: #23795
• Complete the Ascend A5 MoE MXFP stack: #29762 @OrangeRedeng
  • W8A8 MXFP8: #30768 @TallMessiWu
  • W4A8 MXFP4: #30318 @LinyuanLi0046
  • W4A4 MXFP4: #30319 @LinyuanLi0046
• Complete Qwen3.5/A5 coverage:
  !!!looks like collisions here!!!
  • W8A8/W4A8/W4A4 validation and RMSNorm fallback: #32150 @pchengma
  • W8A8 MXFP8: #32266 @TallMessiWu; replaces closed draft #32155
  • W4A8 MXFP4 online and offline Qwen3.5 MoE paths: #32601 @TallMessiWu
  • W4A4 MXFP4 online and offline Qwen3.5 MoE paths: #32602 @TallMessiWu
  • Add the Ascend A5 GemmaRMSNorm fallback required by Qwen3.5: #32745 @TallMessiWu
• W4A4 FRACTAL_NZ linear path: #20860 @OrangeRedeng
• Fix ModelSlim MXFP4 packed-weight loading: #32013 @TallMessiWu
• Fix the W4A8 MoE performance regression: #32113
• Add group-size support for W8A8, W4A8, W4A4, and related schemes.
• Complete Compressed-Tensors FP8 and Quark checkpoint support on Ascend.
• Add QoQ, PDMIX, and new checkpoint formats where backend kernels exist.

The detailed Ascend format/model matrix remains tracked in #21584.

2.4 CPU, Intel XPU, and Apple MLX

• Add MLX on-the-fly Q4/Q8 quantization: #24907
• Add AutoRound AMX CPU support: #29593 @WeiweiZhang1
• Add AWQ/GPTQ INT4 dense linear support on Intel XPU: #30236 @YangKai0616
• Support Compressed-Tensors W4A16 Sparse24: #31248, #31555 @SuperMarioYL
• Make platform-specific quantization validation and documentation complete: #32253
• Add CPU W8A16 per-tensor FP8 GEMM support: #32618 @xinguozhu-2026
• Fix XPU FP8 kernel imports and post-load requantization: #32684 @skavulya

3. Attention and KV-Cache Quantization

3.1 Attention and MLA

• Add a common quantized-attention/MLA scheme abstraction instead of model- and backend-specific branches.
• Integrate Q8KV8 FP8 sparse MLA prefill into the DSA backend: #30514
• Extend Q8KV8 sparse prefill to GLM-5.2 and DeepSeek-V3.2 paths: #31888 @JackChuang
• Load ModelOpt FP4 attention weights and baked FP8 KV scales for Qwen3.5 MoE: #31220 @vroomfondel
• Serve ModelOpt mixed-precision attention and shared experts: #31855
• Support Quark PTPC FP8 attention on ROCm MLA: #28734
• Add FP8 vision attention through FlashAttention-3: #26211 @kousakawang
• Accelerate ROCm unified-attention decode with scaled FP8 Q: #31856 @zijiecode
• Add prefill-only FP8 dense projection GEMMs for MLA q_b/o_proj: #31955 @Raiden-Makoto
• Evaluate 8-bit SageAttention support: #1763

3.2 FP8 KV cache

• Triton attention backend support: #18882
• Fix TRT-LLM MLA with FP8 KV cache and no RoPE: #32181
• Fuse FP8 KV quantization into the store kernel: #31652 @waynehacking8
• Make KV-scale detection and warnings consistent: #31660, #31224
• Resolve the unfused FP8 KV decode overhead: #30815
• Complete Ascend FP8 KV cache beyond MLA-only support: #29641 @LinyuanLi0046
• Support FP8 KV cache on CPU: #32733 @Valentine233; replaces #14482

3.3 INT8 and FP4 KV cache

• Complete Ascend INT8 KV cache and remove the current model/FIA/CANN limitations: #24695 @TamirBaydasov
• Add MX-style FP4 E2M1 KV cache for MHA: #12612
• Add the NVFP4 KV scheme/kernel abstraction and SM120 implementation: #21954, #21601
• Extend NVFP4 KV prefill across SM80/SM90/SM100: #23354 @Tom-Zheng
• Complete SM120/SM121 Gemma FP4 KV and prefix-cache correctness: #29304 @jethac
• Integrate the MXFP4 KV-cache decode kernel for DeepSeek-V4 on Hopper: #32741 @TobyMint
• Finish the umbrella FP4 KV Roadmap #29913

Closed prototypes retained for history:

• #17733 — B200 FP4 KV prototype, closed without merge
• #18314 — SM120 NVFP4 KV prototype, closed without merge

3.4 New KV compression methods

• OSCAR mixed-precision INT2 KV cache: #32129 @zzz0906
• KVarN K4V4/K4V2 KV compression: #27356, #31967 @jtabet
• TurboQuant fused 4-bit KV compression: #23135 @liuhuijiayou
• KVTC calibrated KV compression: #30419, #30417 @michal-mielewczyk-huawei
• Define a unified KV-cache sparsity, capability, and evaluation contract covering attention backends, prefill/decode kernels, prefix caching, speculative decoding, HiCache, and disaggregated transfer: #32657
• Add quantized KV transfer for disaggregated serving. No dedicated implementation PR is currently tracked.

4. Communication Quantization

• Ascend TP communication compression for Qwen3: #20520
• DeepEP low-latency FP8 communication for W4A8: #22316
• ROCm fused all-reduce + RMSNorm + per-group FP8 quantization: #24651
• ROCm fused all-reduce + RMSNorm + per-token FP8/MXFP4 quantization: #29723 @mqhc2020
• ROCm AITER all-reduce + residual RMSNorm + FP8 quantization: #17262 @qichu-yun
• SM120 PCIe all-reduce with fused RMSNorm and FP8 DMA ring: #32479 @AliceChenyy
• Generalize communication quantization beyond individual Qwen3, DeepEP, AITER, and hardware-specific paths.
• Define scale ownership and wire format for TP, EP, CP, all-gather, reduce-scatter, all-reduce, and disaggregated KV transfer.

5. VLM, Diffusion, Speculative Decoding, and LoRA

5.1 VLM and multimodal encoders

• Qwen3.5 block-wise FP8 quantization and model adaptation: #18926
• Complete Ascend Qwen3.5 MXFP8/MXFP4 VLM coverage: #21584, #32266
• Add FP8 vision attention through FlashAttention-3: #26211
• Quantize Gemma 4 MTP bridge projections: #32440 @ayush1399
• Require each production format to declare coverage for the vision/audio encoder, multimodal projector, language trunk, MoE layers, and optional unquantized modules.

5.2 Diffusion transformers

• General quantization plumbing for DiTs: #19156
• Ascend Wan2.2 ModelSlim quantization: #17996
• Ascend Wan2.2 MXFP8 and MXFP4: #20922, #22338
• Ascend Wan2.2 FA MXFP8: #32041 @hxhhhlalala
• ModelOpt FP8 for Flux1/Flux2/Wan2.2: #22365
• ModelOpt NVFP4 for Wan2.2 and Qwen-Image: #22681, #28928
• Fix per-shard FP8 scale shapes for single-GPU fused diffusion linears: #32157
• Fix LTX-2 ModelOpt NVFP4 prefix propagation and scale-layout detection: #32513 @Qyx5-5
• Add Quant-VideoGen PRQ KV-cache quantization for long-horizon causal video generation: #32581 @haochengxi
• Support additional diffusion checkpoint formats, including Compressed-Tensors: #26457
• Complete ROCm diffusion MXFP4 compilation and address the current FP8 GEMM failures: #27702, #29590
• Add recurring image-quality, latency, throughput, memory, and multi-GPU CI for every production diffusion format.

The broader Ascend diffusion work remains tracked in #18967.

5.3 Speculative decoding, LoRA, and disaggregated serving

• Require quantized target and draft compatibility for MTP, EAGLE, DFlash, and other speculative methods.
• Cover quantized attention, KV cache, fused MoE, and bridge projections in speculative models: #32440
• Add LoRA validation for dense and MoE FP8/NVFP4/MXFP4/W4A8 paths.
• Make quantized MoE LoRA CUDA-graph buffer sizing independent of packed WNA16/Marlin tensor layouts: #32704 @yushengsu-thu
• Add disaggregated prefill/decode recipes and CI for production formats: #32120
• Support quantized checkpoints and layout conversion in weight daemons and remote-update workflows: #32335, #32398
• Keep the AutoRound MMLU acceptance threshold aligned with the supported accuracy envelope: #31748

6. New Low-Bit Technologies

• Online rotations for QuaRot-style inference: #5943 @amd-rcorzine
• Ascend ModelSlim QuaRot support for EAGLE3: #30524 @heziiop
• SpinQuant and FlatQuant
• Vector/codebook quantization: QuIP#, AQLM, and VPTQ. The earlier VPTQ PR #3879 was closed without merge.
• OCP MXFP6 and MXINT8
• BitNet-style ternary inference
• Sparse-plus-low-bit execution
  • Compressed-Tensors W4A16 Sparse24: #31248, #31555
• 2-bit and mixed-precision KV cache
  • OSCAR INT2: #32129
  • KVarN K4V2: #31967

7. Active Production Blockers

These open issues should block a “production-ready” claim for the affected combination:

• MiniMax-M3 NVFP4 loading can drop MXFP8 scales or select the wrong MoE method: #31827
• AMX MoE router handling does not enforce FP32 gating logits/weights: #31861
• GLM-5.2 NVFP4 disaggregated prefill can crash in the FlashInfer TRT-LLM MoE path: #31864
• Intermittent CUTLASS NVFP4 GEMM failure on SM121: #32021
• Fused post-quant kernels can generate invalid launch configurations: #32065
• Qwen3.6 NVFP4 checkpoint loading failure: #32156
• Serialized Marlin checkpoints cannot currently be resolved and loaded through the public quantization registry: #32250; fix: #32644
• GLM-5.2 FP4 speculative decoding can select an incompatible BF16 MoE kernel on SM100: #32377
• NVFP4 model output corruption: #32426
• EAGLE can collapse radix-prefix reuse for GLM-DSA NVFP4 workloads: #32459
• DeepSeek-V4 Flash-FP8 can crash with interleaved prefill context parallelism: #32553
• Kimi-K2.6 NVFP4 has a piecewise-prefill CUDA-graph throughput regression: #32655
• Gemma 4 NVFP4 with FP8 KV cache and Frozen-KV MTP can corrupt concurrent requests: #32666
• Mixed-precision Compressed-Tensors checkpoints can silently load completely unquantized: #32736
• Nemotron Nano mixed MXFP8/NVFP4 checkpoints can produce empty output: #32740
• Backend/KV-dtype/platform compatibility is validated inconsistently: #31774; DSA slice: #32576 @mosya415

Related resources

• NPU Quantization Roadmap #14424
• Quantization Modifications Roadmap #15194
• Ascend A5 MXFP8/MXFP4 RFC #21584
• FP4 KV Roadmap #29913
• Weight Loader v2 #31051
• Unified kernel namespace RFC #29630

[MrAnayDongre]

Hi @OrangeRedeng I found one small ambiguity while auditing the quantization registry.
marlin is accepted by --quantization, but it is not a direct key in QUANTIZATION_METHODS. It seems to work only when matching checkpoint metadata resolves it.
Is that intentional?
I’d be happy to contribute a small P0 fix that adds a CLI-to-registry consistency test and corrects the CPU error message to list only CPU_QUANTIZATION_METHODS.

[ping1jing2]

I found one small ambiguity while auditing the quantization registry. marlin is accepted by --quantization, but it is not a direct key in QUANTIZATION_METHODS. It seems to work only when matching checkpoint metadata resolves it. Is that intentional?

i don't think this is expected result. Could you please create another issue to describe the specific example for this question? @OrangeRedeng please help to double-check this scenario.

I’d be happy to contribute a small P0 fix that adds a CLI-to-registry consistency test and corrects the CPU error message to list only CPU_QUANTIZATION_METHODS.

welcome and please go ahead.

[MrAnayDongre]

Thanks. I opened #32250 with a minimal reproduction of the marlin CLI-to-registry mismatch.

[mosya415]

I want to take the DSA slice of 1.3 (also listed under 7) as a first concrete step, continuing #31346 which is merged. The idea: replace the if/elif in _dsa_split_backend_resolution with one declarative table. Each backend declares the cells it supports: platform (sm90, sm100, hip), kv-cache dtype, and phase (prefill or decode, since flashmla_auto is prefill-only, see #31344), plus a priority used for defaulting. The table is the single source for three things. Defaulting picks the highest-priority supported cell. Validation rejects any combination outside the table with one uniform error that lists the valid backends for the given dtype and platform, generated from the table instead of hand-written like in #31346. A parametrized test iterates the full product of axes, so every supported and unsupported cell is asserted, and a backend added without declared cells fails the test by itself.

This is now up as #32576. Defaults are unchanged, the historical branches are pinned as the spec in the unit test over the full input product, rejections are limited to combinations already proven unsupported, and the real cells I can verify on H100. Field names (backend, platform, dtype, phase, priority) are picked so the wider 1.3 metadata (scheme, granularity, kernel) can extend the same shape later. @OrangeRedeng @ping1jing2 if the registry format is planned differently, I will adapt the table to it.

[OrangeRedeng]

I want to take the DSA slice of 1.3 (also listed under 7) as a first concrete step, continuing #31346 which is merged. The idea: replace the if/elif in _dsa_split_backend_resolution with one declarative table. Each backend declares the cells it supports: platform (sm90, sm100, hip), kv-cache dtype, and phase (prefill or decode, since flashmla_auto is prefill-only, see #31344), plus a priority used for defaulting. The table is the single source for three things. Defaulting picks the highest-priority supported cell. Validation rejects any combination outside the table with one uniform error that lists the valid backends for the given dtype and platform, generated from the table instead of hand-written like in #31346. A parametrized test iterates the full product of axes, so every supported and unsupported cell is asserted, and a backend added without declared cells fails the test by itself.

This is now up as #32576. Defaults are unchanged, the historical branches are pinned as the spec in the unit test over the full input product, rejections are limited to combinations already proven unsupported, and the real cells I can verify on H100. Field names (backend, platform, dtype, phase, priority) are picked so the wider 1.3 metadata (scheme, granularity, kernel) can extend the same shape later. @OrangeRedeng @ping1jing2 if the registry format is planned differently, I will adapt the table to it.

Hello! Thanks for your contribution. #32576 looks like a good DSA-scoped first step for §1.3 and addresses the corresponding §7 issue. I’ve updated the roadmap to track your PR and ownership. At the moment, I’m not aware of any plans to use a different registry format. If that changes, we can extend or adapt this structure later.