AWQ Qwen and Phi mappings #1440
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brian-dellabetta
requested review from
kylesayrs,
dsikka,
rahul-tuli and
shanjiaz
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
kylesayrs
pushed a commit
that referenced
this pull request
May 21, 2025
… balance layer input length (#1451) ### Summary We are hitting an edge case in AWQ we had not previously hit with the initial Llama/Qwen testing models. When a smooth layer's # of output_features does not match a balance layer's # of input_features, the code as it is currently will error out when trying to update the smooth layer's weights with `weights.div(scales)`, due to a shape mismatch error. We are hitting this in #1440 for Phi3 models, which include a mapping between the fused `qkv_proj` smooth layer and `o_proj` balance layer in AutoAWQ (see [here](https://github.com/casper-hansen/AutoAWQ/blob/main/awq/models/phi3.py#L51-L57)). The resolution in AutoAWQ is to only use the last rows of the smooth layer so that the shapes line up, as shown [here](https://github.com/casper-hansen/AutoAWQ/blob/main/awq/quantize/scale.py#L123). This PR includes that update, and with #1440 will allow Phi3 models to be quantizable with AWQModifier. Like with v_proj -> o_proj, if shapes don't match up, they will be excluded from resolved mappings. This allows [phi-3-mini](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/tree/main?show_file_info=model-00001-of-00002.safetensors) to include the mapping because `qkv_proj out_features == 3*o_proj in_features == 9216`, but excludes it from [phi-3-medium](https://huggingface.co/microsoft/Phi-3-medium-128k-instruct/tree/main?show_file_info=model-00001-of-00006.safetensors) which has `qkv_proj out_features == 7680` and `o_proj in_features==5120`. If the mapping is included for phi-3-medium, the model blows up with wikitext eval perplexities >2000. This implementation was agreed upon with @anmarques . PS: I also moved `mul` & `div` to `mul_` & `div_`, to avoid unnecessary memory allocation. ------------- ### Test Plan With these changes and with #1440 , `examples/awq/llama_example.py` works with `"microsoft/Phi-3-mini-128k-instruct"` and produces similar results as when qkv_proj to o_proj mapping is included Without mapping: | Tasks |Version|Filter|n-shot| Metric | | Value | |Stderr| |--------|------:|------|-----:|---------------|---|------:|---|------| |wikitext| 2|none | 5|bits_per_byte |↓ | 0.6474|± | N/A| | | |none | 5|byte_perplexity|↓ | 1.5664|± | N/A| | | |none | 5|word_perplexity|↓ |11.0201|± | N/A| With mapping: | Tasks |Version|Filter|n-shot| Metric | | Value | |Stderr| |--------|------:|------|-----:|---------------|---|------:|---|------| |wikitext| 2|none | 5|bits_per_byte |↓ | 0.6482|± | N/A| | | |none | 5|byte_perplexity|↓ | 1.5672|± | N/A| | | |none | 5|word_perplexity|↓ |11.0527|± | N/A| I also confirmed re-running with `meta-llama/Llama-3.2-3B-Instruct` and `meta-llama/Llama-2-7b-hf` does not deviate in PPL scores from what is currently on `main` --------- Signed-off-by: Brian Dellabetta <[email protected]>
brian-dellabetta
force-pushed
the
bdellabe/awq-qwen-mappings
branch
from
56f6069 to
54fa07c
Compare
rahul-tuli
previously approved these changes
May 21, 2025
rahul-tuli
approved these changes
May 21, 2025
kylesayrs
approved these changes
May 21, 2025
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
brian-dellabetta
force-pushed
the
bdellabe/awq-qwen-mappings
branch
from
ad350cb to
32622b1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY:
I wanted to create a PR showing users how they can add more mappings to AWQ to account for more models. Turns out qwen has the exact same as Llama, so I added one for Phi as well. I also updated the naming and used the infer pattern employed in SmoothQuant, rather than requiring user to set it
TEST PLAN:
examples/awq/llama_example.pyworks on this branch forTODOs: