AWQ Modifier

[brian-dellabetta]

SUMMARY:
Addition of AWQModifier, based on AutoAWQ implementation.

Should be reviewed/merged in conjunction with neuralmagic/compressed-tensors#269

Replaces #181 and #824

TEST PLAN:
Some unit tests included, but as this was mostly a port from AutoAWQ, we validated the code by ensuring we could reproduce the evaluation metrics in Table 4 of the paper. We achieve the following wikitext PPL scores:

Llama-2 7B Group 128:

1. Paper: 5.60
2. AutoAWQ: 5.615
3. This implementation: 5.612
4. we match what the paper reports for just RTN -- 5.73
5. We get reasonable results for channel-wise -- 6.788. AutoAWQ errors out for this (setting "q_group_size": -1 in the quant_config), and results not reported in paper.

Llama-2 13B Group 128:

1. We match the results of AutoAWQ and the results shown in the paper: 4.97
2. We match what the paper reports for just RTN -- 4.984

NOTE: We are excluding the clipping logic in this implementation, if we want to add it we should add it as another modifier, they are mutually exclusive and the data model for AWQ doesn't align well with clipping. That might be the reason for the slight deviation of results reported in the paper and in our implementation

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[dsikka]

Should we add evals comparing to GPTQ?

[brian-dellabetta]

Using the latest commit at this time, I am getting the following results via lm-eval.

deepseek-ai/DeepSeek-R1-Distill-Llama-8B:
 dense:
   #gsm flexible-extract, strict-match
   gsm8k: .6619, .6490
   wititext ppl: 15.4498
 awq+quant sym:
   gsm8k: .6376, .6217
   wititext ppl: 18.8623
 quant sym:
   gsm8k: .6732, .6543
   wititext ppl: 16.7398
meta-llama/Llama-2-7b-hf:
 dense:
   gsm8k: .1342, .1342
   wititext ppl: 8.7587
 awq+quant sym:
   gsm8k: .1024, .1001
   wititext ppl: 9.194
 quant sym:
   gsm8k: .1183, .1152
   wititext ppl: 9.311

[brian-dellabetta]

Comparing AWQ vs. GPTQ vs. RTN for meta-llama/Llama-2-7b-hf, using example script:

Type	gsm8k	wikitext
FP16	.1395, .1387	8.7521
AWQ ASYM	.1281, .1274	9.0281
GPTQ ASYM	.1312, .1296	9.1954
AWQ+GPTQ ASYM	.1251, .1221	9.1449
RTN ASYM	.1198, .1190	9.2098
AWQ SYM	.1069, .1054	9.1931
GPTQ SYM	.1046, .1039	9.3525
AWQ+GPTQ SYM	.0955, .0925	9.4326
RTN SYM	.1183, .1152	9.3114

[kylesayrs]

Please remove/fix the example, otherwise LGTM

[dsikka]

Can we fix quality

[Jim2016713]

SUMMARY: Addition of AWQModifier, based on AutoAWQ implementation.

Should be reviewed/merged in conjunction with neuralmagic/compressed-tensors#269

Replaces #181 and #824

TEST PLAN: Some unit tests included, but as this was mostly a port from AutoAWQ, we validated the code by ensuring we could reproduce the evaluation metrics in Table 4 of the paper. We achieve the following wikitext PPL scores:

Llama-2 7B Group 128:

1. Paper: 5.60
2. AutoAWQ: 5.615
3. This implementation: 5.612
4. we match what the paper reports for just RTN -- 5.73
5. We get reasonable results for channel-wise -- 6.788. AutoAWQ errors out for this (setting "q_group_size": -1 in the quant_config), and results not reported in paper.

Llama-2 13B Group 128:

1. We match the results of AutoAWQ and the results shown in the paper: 4.97
2. We match what the paper reports for just RTN -- 4.984

NOTE: We are excluding the clipping logic in this implementation, if we want to add it we should add it as another modifier, they are mutually exclusive and the data model for AWQ doesn't align well with clipping. That might be the reason for the slight deviation of results reported in the paper and in our implementation

awq支持qwen2.5 7b模型的量化吗，我使用main分支的代码进行量化，报了这样的错
RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 2048 but got size 1874 for tensor number 181 in the list.

[brian-dellabetta]

awq支持qwen2.5 7b模型的量化吗，我使用main分支的代码进行量化，报了这样的错 RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 2048 but got size 1874 for tensor number 181 in the list.

@Jim2016713 if you'd like to report a bug or issue please create a ticket with the corresponding details the form requests -- env versions, code snippet, full stack trace.

[ljwh]

awq支持qwen2.5 7b模型的量化吗，我使用main分支的代码进行量化，报了这样的错 RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 2048 but got size 1874 for tensor number 181 in the list.

@Jim2016713 the error msg means you should padding to the max length in tokenize function, i meet the same problem and solve like this:

```tokenizer(..., max_length=some_number, padding="max_length", truncation=True)

[brian-dellabetta]

Thanks @ljwh , this should be resolved on main as part of #1426 , though we haven't created a new version just yet.

[ljwh]

Thanks @ljwh , this should be resolved on main as part of #1426 , though we haven't created a new version just yet.

Is there a release plan? seems a lot of fix of awq memory use

[brian-dellabetta]

Thanks @ljwh , this should be resolved on main as part of #1426 , though we haven't created a new version just yet.

Is there a release plan? seems a lot of fix of awq memory use

@ljwh we will cut a release soon, a couple more fixes are in transit, #1435 & #1444