Repository navigation
Dev zyq - #8
Merged
Merged
Dev zyq#8
Conversation
… INT8 path
"re:.*mlp.gate.*" only exists to protect the MoE router, but its trailing
".*" (plus the unescaped dot) also matched the dense mlp.gate_proj. That
weight stayed in BF16 while up_proj was INT8, so SGLang/vLLM refused the
fused gate_up_proj with
ValueError: Found different quantization schemes for ['gate_proj', 'up_proj']
and the artifact was only partially quantized (5.7B params kept BF16).
Anchor the rule and cover both spellings: the engine matches module names
("...mlp.gate") while this converter matches state-dict keys
("...mlp.gate.weight").
Verified on Qwen3.8-27B: same key set as the released
Qwen3.8-27B-Channel-INT8-w8a8 artifact (1606 keys) and identical
quantization decisions for all 937 Linear weights; the MoE router is still
ignored, so Qwen3.5-397B-A17B is unaffected.
AI Review审查尚未完整完成，无法给出整体结论；当前没有可发布的已确认问题，不代表代码没有问题。
|
![hygon-ai-reviewer[bot]](https://avatars.githubusercontent.com/u/280652917?s=60&v=4){kind=small}
There was a problem hiding this comment.
AI Review
🟢 未发现有明确证据的问题
在已提供并完成审查的 diff 范围内未形成可确认问题。
变更概览
本次变更主要包含：
- 将量化模板中 MoE 路由层的忽略规则由过宽的 `mlp.gate` 前后通配匹配收窄为精确匹配 `mlp.gate` 与 `mlp.gate.weight`，避免在 dense 模型上连带命中 gate_proj 导致融合分片量化方案不一致。
文件审查摘要
| 文件 | 变更 | 审查结果 |
|---|---|---|
templates/qwen3_5_bf16_to_channel.py |
修改 · +2/-1 | — |
审查信息
- 变更统计：1 个文件，+2/-1。
- 覆盖情况：共 1 个文件，已完整审查 1 个。
- 候选问题：0 项；证据复核过滤：0 项；发布前敏感信息保护：0 项。
- 本服务以 GitHub 提供的 PR diff 为审查主体，PR 描述与按相关性选取的仓库片段仅用于核验；未执行代码或重跑测试，结论仍需维护者核验。
caihl706
approved these changes
Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1."re:.mlp.gate.",它的本意是保护 MoE 的路由层
mlp.gate（Qwen3_5MoeTopKRouter，选专家的门控），让路由层保持 BF16 不量化。但这条规则的写法过于宽泛，在 dense 模型（如Qwen3.8-27B，Architecture =Qwen3_5ForConditionalGeneration）上会连带命中普通 MLP 的mlp.gate_proj，这个模块是FP16，而mlp.up_proj.weight是INT8的，sglang的gate_up_proj融合模块失败，融合模块的两个分片方案不一致，加载失败2.用该模板脚本量化dense模型Qwen3.8-27B为INT8,w8a8时，slang启动失败,改为更精确的匹配规则，让
mlp.gate_proj模块恢复量化，sglang启动正常，ppl测试正确