Skip to content

[Bugfix] Fix shape calculation for group quantization #308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 2, 2025

Conversation

kylesayrs
Copy link
Contributor

Purpose

  • Fix shape calculation for group quantization
  • This was uncovered while attempting to quantize the vision tower of qwen2.5 model. This likely went unnoticed, since most model shapes are powers of two, something that is not true of weights in the qwen 2.5 vision tower

Changes

  • Use the ceil of num columns / group_size in order to reflect that the remainder columns are a group

Testing

Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
@kylesayrs kylesayrs changed the title [Bugfix] Use ceil [Bugfix] Fix shape calculation for group quantization Apr 29, 2025
Copy link
Collaborator

@dsikka dsikka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should verify effects on vLLM, if any as I think the same calculation is assumed there when creating parameters for weight loading

@kylesayrs
Copy link
Contributor Author

@dsikka That seems to be the case
https://github.com/vllm-project/vllm/blob/1c2bc7ead019cdf5b04b2f1d07b00982352f85ef/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_wNa16.py#L104-L108

I can only assume that this is a mistake on vLLM side too that hasn't been triggered, in the same way that it hasn't been an issue for LC

@rahul-tuli
Copy link
Member

Approved pending we raise a warning on llmcompressor side, for incompatible shapes with vllm. Thanks for the diff

@kylesayrs kylesayrs enabled auto-merge (squash) May 2, 2025 16:34
@kylesayrs kylesayrs requested a review from dsikka May 2, 2025 16:35
@kylesayrs kylesayrs merged commit 6148fef into main May 2, 2025
1 check passed
@kylesayrs kylesayrs deleted the kylesayrs/fix-group-bug branch May 2, 2025 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants