Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we have pushed an update to the development branch (and the Triton backend) this Feb 25, 2025.

This update includes:

• Model Support
  • Added Mamba-Hybrid support.
• Features
  • AutoDeploy workflow.
    • The AutoDeploy workflow is an experimental feature in tensorrt_llm._torch.auto_deploy. AutoDeploy provides an automated path from off-the-shelf models to optimized deployment in the TensorRT-LLM runtime. Check out examples/auto_deploy/README.md for more details.
  • Added NVILA video support. The support includes 1 prompt - N media and N prompt - N media batching modes.
  • Added a --quantize_lm_head option examples/quantization/quantize.py to support lm_head quantization.
  • LLM API
    • Added abort request support.
    • Added get_stats support.
• Bug fixes
  • Fixed a Llama-3.2 SmoothQuant convert checkpoint issue. (Llama-3.2 SmoothQuant convert checkpoint error #2677)
  • Fixed a bug when loading an engine using LoRA through the LLM API. (Bug when loading an engine using LoRA through LLM API #2782)
• Known Issues
  • The PyTorch workflow on SBSA is incompatible with bare metal environments like Ubuntu 24.04. Please use the [PyTorch NGC Container (https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) for optimal support on SBSA platforms.

Thanks,
The TensorRT-LLM Engineering Team