[Feature]: support for nvidia/Qwen2.5-VL-7B-Instruct-FP4

[pythonjavaerlang]

🚀 The feature, motivation and pitch

Nvidia L4 (25 GB)
https://huggingface.co/nvidia/Qwen2.5-VL-7B-Instruct-FP4
nvidia-smi
| NVIDIA-SMI 550.163.01 Driver Version: 535.261.03 CUDA Version: 13.0 |

python3 ./TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py --model_dir ./Qwen2.5-VL-7B-Instruct-FP4 --output_dir ./tllm_checkpoint_fp4 --dtype float16 --smoothquant 0.5

[TensorRT-LLM] TensorRT LLM version: 1.2.0rc1
1.2.0rc1
Traceback (most recent call last):
File "/code/tensorrt_llm/./TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py", line 345, in
main()
File "/code/tensorrt_llm/./TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py", line 337, in main
convert_and_save_hf(args)
File "/code/tensorrt_llm/./TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py", line 263, in convert_and_save_hf
QWenForCausalLM.quantize(
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/qwen/model.py", line 530, in quantize
config = QWenConfig.from_hugging_face(hf_model_dir,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/qwen/config.py", line 115, in from_hugging_face
assert qwen_type in valid_types, f"Unsupported Qwen type: {qwen_type}, only {valid_types} are acceptable."
^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Unsupported Qwen type: qwen2_5_vl, only ('qwen', 'qwen2', 'qwen2_moe', 'qwen2_llava_onevision', 'qwen2_vl', 'qwen2_audio', 'qwen3', 'qwen3_moe') are acceptable.

uname -a
Linux f909e01d9262 6.12.38+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.38-1 (2025-07-16) x86_64 x86_64 x86_64 GNU/Linux

Alternatives

When I edit code adding model name manually, I receive this error: [TensorRT-LLM] TensorRT LLM version: 1.2.0rc1
1.2.0rc1
torch_dtype is deprecated! Use dtype instead!
Traceback (most recent call last):
File "/code/tensorrt_llm/./TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py", line 345, in
main()
File "/code/tensorrt_llm/./TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py", line 337, in main
convert_and_save_hf(args)
File "/code/tensorrt_llm/./TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py", line 263, in convert_and_save_hf
QWenForCausalLM.quantize(
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/qwen/model.py", line 535, in quantize
convert.quantize(hf_model_dir,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/qwen/convert.py", line 996, in quantize
hf_model = model_cls.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 607, in from_pretrained
raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers.models.qwen2_5_vl.configuration_qwen2_5_vl.Qwen2_5_VLConfig'> for this kind of AutoModel: Au
toModelForCausalLM.
Model type should be one of ApertusConfig, ArceeConfig, AriaTextConfig, BambaConfig, BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegas
usConfig, BioGptConfig, BitNetConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, LlamaConfig, CodeGenConfig, CohereConfig, Cohere2
Config, CpmAntConfig, CTRLConfig, Data2VecTextConfig, DbrxConfig, DeepseekV2Config, DeepseekV3Config, DiffLlamaConfig, DogeConfig, Dots1Config, ElectraConfig,
Emu3Config, ErnieConfig, Ernie4_5Config, Ernie4_5_MoeConfig, Exaone4Config, FalconConfig, FalconH1Config, FalconMambaConfig, FuyuConfig, GemmaConfig, Gemma2C
onfig, Gemma3Config, Gemma3TextConfig, Gemma3nConfig, Gemma3nTextConfig, GitConfig, GlmConfig, Glm4Config, Glm4MoeConfig, GotOcr2Config, GPT2Config, GPT2Confi
g, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GptOssConfig, GPTJConfig, GraniteConfig, GraniteMoeConfig, GraniteMoeHybridConfig, Gr
aniteMoeSharedConfig, HeliumConfig, HunYuanDenseV1Config, HunYuanMoEV1Config, JambaConfig, JetMoeConfig, Lfm2Config, LlamaConfig, Llama4Config, Llama4TextConf
ig, MambaConfig, Mamba2Config, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MiniMaxConfig, MistralConfig, MixtralConfig, MllamaConfig, ModernBer
tDecoderConfig, MoshiConfig, MptConfig, MusicgenConfig, MusicgenMelodyConfig, MvpConfig, NemotronConfig, OlmoConfig, Olmo2Config, OlmoeConfig, OpenLlamaConfig
, OpenAIGPTConfig, OPTConfig, PegasusConfig, PersimmonConfig, PhiConfig, Phi3Config, Phi4MultimodalConfig, PhimoeConfig, PLBartConfig, ProphetNetConfig, QDQBe
rtConfig, Qwen2Config, Qwen2MoeConfig, Qwen3Config, Qwen3MoeConfig, RecurrentGemmaConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormCon
fig, RoCBertConfig, RoFormerConfig, RwkvConfig, SeedOssConfig, SmolLM3Config, Speech2Text2Config, StableLmConfig, Starcoder2Config, TransfoXLConfig, TrOCRConf
ig, WhisperConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, xLSTMConfig, XmodConfig, ZambaConfig, Zamba2
Config.

Additional context

The following document says Qwen2.5 is supported:
https://nvidia.github.io/TensorRT-LLM/reference/support-matrix.html#models-pytorch-backend
"""
Qwen2_5_VLForConditionalGeneration | Qwen2.5-VL | Qwen/Qwen2.5-VL-7B-Instruct
"""

Docker image:
nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc0.post1

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[karljang]

@pythonjavaerlang ,
This checkpoint Qwen2.5-VL-7B-Instruct-FP4 isn’t compatible with L4 GPUs. FP4 checkpoints require Blackwell or newer hardware.

In the meantime, you can try running it using PyTorch backend using this command:

trtllm-serve Qwen/Qwen2.5-VL-7B-Instruct --backend pytorch

And, if you're looking for a smaller model, consider using Qwen/Qwen2.5-VL-3B-Instruct model.

[yechank-nvidia]

Hi, the bugfix PR has been submitted: #8680

[yechank-nvidia]

Can you try on ToT main? As #8680 is merged into the main, you can try on ToT main.

[HugoLifes]

mmm im trying in blackwell's b200 and this is the print

trtllm-serve serve nvidia/Qwen2.5-VL-7B-Instruct-FP4
--backend pytorch
--host 0.0.0.0
--port 8000
2025-10-29 05:58:08,688 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[10/29/2025-05:58:09] [TRT-LLM] [I] Compute capability: (10, 0)
[10/29/2025-05:58:09] [TRT-LLM] [I] SM count: 148
[10/29/2025-05:58:09] [TRT-LLM] [I] SM clock: 1965 MHz
[10/29/2025-05:58:09] [TRT-LLM] [I] int4 TFLOPS: 0
[10/29/2025-05:58:09] [TRT-LLM] [I] int8 TFLOPS: 0
[10/29/2025-05:58:09] [TRT-LLM] [I] fp8 TFLOPS: 0
[10/29/2025-05:58:09] [TRT-LLM] [I] float16 TFLOPS: 0
[10/29/2025-05:58:09] [TRT-LLM] [I] bfloat16 TFLOPS: 0
[10/29/2025-05:58:09] [TRT-LLM] [I] float32 TFLOPS: 0
[10/29/2025-05:58:09] [TRT-LLM] [I] Total Memory: 179 GiB
[10/29/2025-05:58:09] [TRT-LLM] [I] Memory clock: 3996 MHz
[10/29/2025-05:58:09] [TRT-LLM] [I] Memory bus width: 7680
[10/29/2025-05:58:09] [TRT-LLM] [I] Memory bandwidth: 7672 GB/s
[10/29/2025-05:58:09] [TRT-LLM] [I] NVLink is active: True
[10/29/2025-05:58:09] [TRT-LLM] [I] NVLink version: 5
[10/29/2025-05:58:09] [TRT-LLM] [I] NVLink bandwidth (unidirectional): 900 GB/s
[10/29/2025-05:58:09] [TRT-LLM] [I] Set nccl_plugin to None.
Unknown quantization type, got modelopt - supported types are: ['awq', 'bitsandbytes_4bit', 'bitsandbytes_8bit', 'gptq', 'aqlm', 'quanto', 'quark', 'eetq', 'higgs', 'hqq', 'compressed-tensors', 'fbgemm_fp8', 'torchao', 'bitnet', 'vptq', 'spqr', 'fp8']. Hence, we will skip the quantization. To remove the warning, you can delete the quantization_config attribute in config.json
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with model.to('cuda').
Traceback (most recent call last):
File "/usr/local/bin/trtllm-serve", line 8, in
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1161, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/serve.py", line 225, in serve
launch_server(host, port, llm_args)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/serve.py", line 90, in launch_server
llm = PyTorchLLM(**llm_args)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/llm.py", line 27, in init
super().init(model, tokenizer, tokenizer_mode, skip_tokenizer_init,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 176, in init
raise e
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 171, in init
self._build_model()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 529, in _build_model
self.input_processor = create_input_processor(self._hf_model_dir,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/inputs/registry.py", line 116, in create_input_processor
return input_processor_cls(model_path_or_dir, model_config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/modeling_qwen2vl.py", line 31, in init
model = self.get_model_class().from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 279, in _wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 4342, in from_pretrained
model = cls(config, *model_args, **model_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1481, in init
super().init(config)
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 1884, in init
self.generation_config = GenerationConfig.from_model_config(config) if self.can_generate() else None
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/generation/configuration_utils.py", line 1290, in from_model_config
decoder_config_dict = decoder_config.to_dict()
^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'to_dict'

[yechank-nvidia]

Hi @HugoLifes, are you doing on main ToT? When I checked with 00eaf5f8837931c74208705869a3e6e6bdae1486 this commit id, it was working alright.

[HugoLifes]

thnks bud so i need the main version o i can do it with pre build docker version container

the image im using is nvcr.io/nvidia/tensorrt-llm/release:latest and TensorRT-LLM v0.20.0 (NGC Pre-built)

[yechank-nvidia]

Yes. Or you can wait for our next release. I will close this for now, and feel free to re-open it when you encounter it again.