diff --git a/docs/source/3x/PT_MXQuant.md b/docs/source/3x/PT_MXQuant.md index 44dbd05e7d4..3472aabea44 100644 --- a/docs/source/3x/PT_MXQuant.md +++ b/docs/source/3x/PT_MXQuant.md @@ -70,7 +70,7 @@ Neural Compressor seamlessly applies the MX data type to post-training quantizat -The memory and computational limits of LLMs are more severe than other general neural networks, so our exploration focuses on LLMs first. The following table shows the basic MX quantization recipes in Neural Compressor and enumerates distinctions among various data types. The MX data type replaces general float scale with powers of two to be more hardware-friendly. It adapts a granularity falling between per-channel and per-tensor to balance accuracy and memory consumption. +The memory and computational limits of LLMs are more severe than other general neural networks, so our exploration focuses on LLMs first. The following table shows the basic MX quantization recipes in Neural Compressor and enumerates distinctions among various data types. The MX data type replaces general float scale with powers of two to be more hardware-friendly. | | MX Format | INT8 | FP8 | |------------|--------------|------------|------------| @@ -78,24 +78,51 @@ The memory and computational limits of LLMs are more severe than other general n | Zero point | 0 (None) | $2^{bits - 1}$ or $-min * scale$ | 0 (None) | | Granularity | per-block (default blocksize is 32) | per-channel or per-tensor | per-channel or per-tensor | -The exponent (exp) is equal to torch.floor(torch.log2(amax)), MAX is the representation range of the data type, amax is the max absolute value of per-block tensor, and rmin is the minimum value of the per-block tensor. +The exponent (exp) is equal to clamp(floor(log2(amax)) - maxExp, -127, 127), MAX is the representation range of the data type, amax is the max absolute value of per-block tensor, and rmin is the minimum value of the per-block tensor. ## Get Started with Microscaling Quantization API -To get a model quantized with Microscaling Data Types, users can use the Microscaling Quantization API as follows. +To get a model quantized with Microscaling Data Types, users can use the AutoRound Quantization API as follows. ```python -from neural_compressor.torch.quantization import MXQuantConfig, prepare, convert - -quant_config = MXQuantConfig(w_dtype=args.w_dtype, act_dtype=args.act_dtype, weight_only=args.woq) -user_model = prepare(model=user_model, quant_config=quant_config) -user_model = convert(model=user_model) +from neural_compressor.torch.quantization import AutoRoundConfig, prepare, convert +from transformers import AutoModelForCausalLM, AutoTokenizer + +fp32_model = AutoModelForCausalLM.from_pretrained( + "facebook/opt-125m", + device_map="auto", +) +tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m", trust_remote_code=True) +output_dir = "./saved_inc" + +# quantization configuration +quant_config = AutoRoundConfig( + tokenizer=tokenizer, + nsamples=32, + seqlen=32, + iters=20, + scheme="MXFP4", # MXFP4, MXFP8 + export_format="auto_round", + output_dir=output_dir, # default is "temp_auto_round" +) + +# quantize the model and save to output_dir +model = prepare(model=fp32_model, quant_config=quant_config) +model = convert(model) + +# loading +model = AutoModelForCausalLM.from_pretrained(output_dir, torch_dtype="auto", device_map="auto") + +# inference +text = "There is a girl who likes adventure," +inputs = tokenizer(text, return_tensors="pt").to(model.device) +print(tokenizer.decode(model.generate(**inputs, max_new_tokens=10)[0])) ``` ## Examples -- PyTorch [huggingface models](/examples/pytorch/nlp/huggingface_models/language-modeling/quantization/mx_quant) +- PyTorch [LLM/VLM models](/examples/pytorch/multimodal-modeling/quantization/auto_round/llama4) ## Reference