[Docs] Add GPTQModel #14056
[Docs] Add GPTQModel #14056
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
|
|
||
| # GPTQModel | ||
|
|
||
| To create a new [2, 3, 4, 8]-bit GPTQ quantized model, you can leverage [GPTQModel](https://github.com/ModelCloud/GPTQModel) from ModelCloud.AI. |
There was a problem hiding this comment.
I was curious about this earlier and didn't get to ask - how are the 2bit and 3bit layers running in vLLM? IIRC we don't have GPTQ kernels for these bits
There was a problem hiding this comment.
Done. Frankly, we never tested 2,3 bits with vllm so made bad assumption there would be a fallback kernel that is 2,3 compatible. We only started testing 2,3 bits in gptqmodel recently as well due to the deepseek needs. Before deepseek, no one gambled with 2/3 bits. =)
There was a problem hiding this comment.
@mgoin Actually, exllama vllm kernel internal code has 2,3 bits support but it was never tested and validated. I plan to adapt this kernel for GPTQModel as well so HF Transformer have access to this kernel. I will do some accuracy comparision vs reference Torch and Marlin and verify 2-8 bits and get back to you. If good, we can just advertise 2,3.
There was a problem hiding this comment.
Make sure to add link in docs/source/features/quantization/index.md
| To run an GPTQModel quantized model with vLLM, you can use [DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2](https://huggingface.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2) with the following command: | ||
|
|
||
| ```console | ||
| python examples/offline_inference/llm_engine_example.py --model DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2 --quantization gptq |
There was a problem hiding this comment.
Please remove --quantization gptq as this with prevent marlin or machete from being used
| sampling_params = SamplingParams(temperature=0.6, top_p=0.9) | ||
|
|
||
| # Create an LLM. | ||
| llm = LLM(model="DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2", quantization="gptq") |
| @@ -12,6 +12,7 @@ supported_hardware | |||
| auto_awq | |||
| bnb | |||
| gguf | |||
| gptqmodel | |||
There was a problem hiding this comment.
@mgoin This list appears to be doing a-z order but fp8 is near the bottom. I would actually re-order this in the performance and level of vllm kernel support, not quant name. bnb and gguf should be near the bottom here but not my decision to make.
| @@ -3,7 +3,7 @@ | |||
| # AutoAWQ | |||
|
|
|||
| To create a new 4-bit quantized model, you can leverage [AutoAWQ](https://github.com/casper-hansen/AutoAWQ). | |||
| Quantizing reduces the model's precision from FP16 to INT4 which effectively reduces the file size by ~70%. | |||
| Quantization reduces the model's precision from BF16/FP16 to INT4 which effectively reduces the total model memory footprint. | |||
There was a problem hiding this comment.
@mgoin GPTQModel doc is based off the awq doc template but I need to fix this sentence.
BF16 to INT4 does not yield 70% model reduction. This is absolute best case scenario and overall model memory does not drop 70%. Depending on params, BPW can approach 5bits just like GPTQ.
PR Changes:
@mgoin