-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add GPTQModel support for evaluating GPTQ models #2217
Add GPTQModel support for evaluating GPTQ models #2217
Conversation
@baberabb Hi, can we get some action on this? What do we need to do to get this reviewed and merged? |
@baberabb Ruff/Lint checks passed. Awaiting review. Thanks. |
@baberabb Ping. Please check our PR. We will push a unit test into |
@baberabb Unit test added. |
Hi! Thanks for the PR, and sorry it took ages for us to review. This looks good to me, but I want to run it through @haileyschoelkopf as well. ps. 3.8 test failing because of a recent |
The error "TypeError: 'type' object is not subscriptable" occurs because the code uses Python 3.9+ type hinting syntax, such as list[str], which is not supported in Python 3.8 |
@baberabb transformers (as far as the new releases) are only validated/ci tested on python 3.9. I think lm eval CI and setup needs to upgrade to depend on min 3.9 Or apply the same patch used by @CL-ModelCloud to all other tests. eb10efb |
Add option to use GPTQModel for
lm_eval
. GPTQModel is a replacement for AutoGPTQ on GPTQ quantization and inference with better model support and much faster inference speed out-of-the-box. We have been using it internally with lm-eval for months without issue.