Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GPTQModel support for evaluating GPTQ models #2217

Merged
merged 25 commits into from
Oct 31, 2024

Conversation

Qubitium
Copy link
Contributor

@Qubitium Qubitium commented Aug 16, 2024

Add option to use GPTQModel for lm_eval. GPTQModel is a replacement for AutoGPTQ on GPTQ quantization and inference with better model support and much faster inference speed out-of-the-box. We have been using it internally with lm-eval for months without issue.

@CLAassistant
Copy link

CLAassistant commented Aug 16, 2024

CLA assistant check
All committers have signed the CLA.

@Qubitium
Copy link
Contributor Author

@baberabb Hi, can we get some action on this? What do we need to do to get this reviewed and merged?

@Qubitium
Copy link
Contributor Author

@baberabb Ruff/Lint checks passed. Awaiting review. Thanks.

@Qubitium
Copy link
Contributor Author

@baberabb Ping. Please check our PR. We will push a unit test into test/modes/test_gptq.py later today to complete the PR. Let us know if there is anything else required of us.

@Qubitium Qubitium changed the title Add GPTQModel support for inferencing GPTQ models Add GPTQModel support for evaluating GPTQ models Oct 24, 2024
@Qubitium
Copy link
Contributor Author

@baberabb Unit test added.

@baberabb
Copy link
Contributor

Hi! Thanks for the PR, and sorry it took ages for us to review. This looks good to me, but I want to run it through @haileyschoelkopf as well.

ps. 3.8 test failing because of a recent transformers update.

@CL-ModelCloud
Copy link
Contributor

Hi! Thanks for the PR, and sorry it took ages for us to review. This looks good to me, but I want to run it through @haileyschoelkopf as well.

ps. 3.8 test failing because of a recent transformers update.

/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/transformers/pipelines/audio_utils.py:54: in <module>
    ffmpeg_additional_args: Optional[list[str]] = None,
E   TypeError: 'type' object is not subscriptable
        Optional   = typing.Optional
        Tuple      = typing.Tuple
        Union      = typing.Union
        __builtins__ = <builtins>
        __cached__ = '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/transformers/pipelines/__pycache__/audio_utils.cpython-38.pyc'
        __doc__    = None
        __file__   = '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/transformers/pipelines/audio_utils.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7fd6d460cc70>
        __name__   = 'transformers.pipelines.audio_utils'
        __package__ = 'transformers.pipelines'
        __spec__   = ModuleSpec(name='transformers.pipelines.audio_utils', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7fd6d460cc70>, origin='/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/transformers/pipelines/audio_utils.py')
        datetime   = <module 'datetime' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/datetime.py'>
        ffmpeg_read = <function ffmpeg_read at 0x7fd6d467f160>
        np         = <module 'numpy' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/numpy/__init__.py'>
        platform   = <module 'platform' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/platform.py'>
        subprocess = <module 'subprocess' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/subprocess.py'>
__________________ ERROR collecting tests/models/test_gguf.py __________________

The error "TypeError: 'type' object is not subscriptable" occurs because the code uses Python 3.9+ type hinting syntax, such as list[str], which is not supported in Python 3.8

Transformers is tested on Python 3.9+

@Qubitium
Copy link
Contributor Author

Qubitium commented Oct 25, 2024

@baberabb transformers (as far as the new releases) are only validated/ci tested on python 3.9. I think lm eval CI and setup needs to upgrade to depend on min 3.9

Or apply the same patch used by @CL-ModelCloud to all other tests. eb10efb

@baberabb baberabb merged commit 4f8e479 into EleutherAI:main Oct 31, 2024
8 checks passed
@Qubitium Qubitium deleted the MOD-Support-GPTQModel branch November 1, 2024 01:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants