feat: extend moe model check to multimodal ones and add block quantization parameters - #605
Conversation
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 11 |
| Duplication | 0 |
TIP This summary will be updated as you push new changes. Give us feedback
llcnt
changed the title
gsprochette
left a comment
gsprochette
left a comment
There was a problem hiding this comment.
Cool PR, there a missing behavior when setting one block_quant_shape and not the other, after this and a quick documentation it will be ready to merge :)
sharpenb
left a comment
sharpenb
left a comment
There was a problem hiding this comment.
Thanks! It looks good. Left a comment to clarify the docs and it is clear for people what to expect :)
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 06b5a61. Configure here.
gsprochette
left a comment
gsprochette
left a comment
There was a problem hiding this comment.
Thanks a lot about the updates in moe_kernel_tuner, much clearer. I think there's a problem in reduce_noe that was introduced so I'll let you address it before I can approve
| config_json[target_name] = smash_config["num_experts_per_token"] | ||
| text_config_json[target_name] = smash_config["num_experts_per_token"] | ||
| with config_path.open("w", encoding="utf-8") as f: | ||
| json.dump(config_json, f, indent=2) |
There was a problem hiding this comment.
Should this be text_config_json instead? I think text_config_json is not being used at all here, have you tested it?
gsprochette
left a comment
gsprochette
left a comment
There was a problem hiding this comment.
All good on my side ! Thanks a lot for the clarifications :)
3 participants
Description
Just extend the model check, and the logic inside the
moe_kernel_tuner.py, so that one can tune multimodel MoE (eg. model that contains a vision tower and a text model nested, as qwen3.5-32b).And make the block quant shape (ie the size of the block on which the scales are computed for re-normalization before quantization) usable from the user.
Related Issue
Fixes #(issue number)
Type of Change
Testing
uv run pytest -m "cpu and not slow")For full setup and testing instructions, see the Contributing Guide.
Checklist