Skip to content

Latest commit

 

History

History
61 lines (48 loc) · 2.7 KB

exllama_v2.md

File metadata and controls

61 lines (48 loc) · 2.7 KB

ExllamaV2 GPTQ Inference Franework

Integrated ExllamaV2 customized kernel into Fastchat to provide Faster GPTQ inference speed.

Note: Exllama not yet support embedding REST API.

Install ExllamaV2

Setup environment (please refer to this link for more details):

git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install -e .

Chat with the CLI:

python3 -m fastchat.serve.cli \
    --model-path models/vicuna-7B-1.1-GPTQ-4bit-128g \
    --enable-exllama

Start model worker:

# Download quantized model from huggingface
# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/TheBloke/vicuna-7B-1.1-GPTQ-4bit-128g models/vicuna-7B-1.1-GPTQ-4bit-128g

# Load model with default configuration (max sequence length 4096, no GPU split setting).
python3 -m fastchat.serve.model_worker \
    --model-path models/vicuna-7B-1.1-GPTQ-4bit-128g \
    --enable-exllama

#Load model with max sequence length 2048, allocate 18 GB to CUDA:0 and 24 GB to CUDA:1.
python3 -m fastchat.serve.model_worker \
    --model-path models/vicuna-7B-1.1-GPTQ-4bit-128g \
    --enable-exllama \
    --exllama-max-seq-len 2048 \
    --exllama-gpu-split 18,24

Performance

Reference: https://github.com/turboderp/exllamav2#performance

Model Mode Size grpsz act V1: 3090Ti V1: 4090 V2: 3090Ti V2: 4090
Llama GPTQ 7B 128 no 143 t/s 173 t/s 175 t/s 195 t/s
Llama GPTQ 13B 128 no 84 t/s 102 t/s 105 t/s 110 t/s
Llama GPTQ 33B 128 yes 37 t/s 45 t/s 45 t/s 48 t/s
OpenLlama GPTQ 3B 128 yes 194 t/s 226 t/s 295 t/s 321 t/s
CodeLlama EXL2 4.0 bpw 34B - - - - 42 t/s 48 t/s
Llama2 EXL2 3.0 bpw 7B - - - - 195 t/s 224 t/s
Llama2 EXL2 4.0 bpw 7B - - - - 164 t/s 197 t/s
Llama2 EXL2 5.0 bpw 7B - - - - 144 t/s 160 t/s
Llama2 EXL2 2.5 bpw 70B - - - - 30 t/s 35 t/s
TinyLlama EXL2 3.0 bpw 1.1B - - - - 536 t/s 635 t/s
TinyLlama EXL2 4.0 bpw 1.1B - - - - 509 t/s 590 t/s