v0.20.0rc0 #3812

kaiyux announced in Announcements

kaiyux
Apr 23, 2025
Maintainer

Highlights

Model Support
- Added Nemotron-H model support (feat: Nemotron-H model support #3430)
- Added Dynasor-CoT in scaffolding examples (feat: Add Dynasor-CoT in scaffolding examples #3501)
Features
- Added stream generation task scaffolding examples (feat: Add stream generation task scaffolding examples #3527)
- Added unfused RoPE support in MLA (feat: Support unfused rope in MLA. #3610)
- Multimodal models
  - Added support in trtllm-serve (feat: trtllm-serve multimodal support #3590)
  - Added support in trtllm-bench, the support is limited to image only for now (feat: adding multimodal (only image for now) support in trtllm-bench #3490)
- (experimental) The TensorRT-LLM Triton backend has supported the LLM API (Update TensorRT-LLM backend triton-inference-server/tensorrtllm_backend#742)
Performance
- Optimized Large Embedding Tables in Multimodal Models (feat: Offloading Multimodal embedding table to CPU in Chunked Prefill Mode #3380)
Infra
- Dependent datasets version was upgraded to 3.1.0 (feat: adding multimodal (only image for now) support in trtllm-bench #3490)

What's Changed

chore: Unify Python NVTX call by @kaiyux in chore: Unify Python NVTX call #3450
doc: genai-perf benchmark & slurm multi-node for trtllm-serve doc by @LinPoly in doc: genai-perf benchmark & slurm multi-node for trtllm-serve doc #3407
fix: disable KV cache reuse if using attention sink by @Funatiq in fix: disable KV cache reuse if using attention sink #3021
doc: Minor fixes for documents by @kaiyux in doc: Minor fixes for documents #3577
fix: nvbugs/5075538: fix cross attention mask when decoder input len > 1 by @VALLIS-NERIA in fix: nvbugs/5075538: fix cross attention mask when decoder input len > 1 #3585
chore: Mass integration of release/0.18 by @dcampora in chore: Mass integration of release/0.18 #3421
fix: LLM API _hf_model_dir for non-cached case by @syuoni in fix: LLM API _hf_model_dir for non-cached case #3562
ci: waive test_llm_multi_node_pytorch by @Superjomn in ci: waive test_llm_multi_node_pytorch #3592
fix: amend trtllm-bench command in the test by @Superjomn in fix: amend trtllm-bench command in the test #3563
feat: Add stream generation task scaffolding examples by @narutolhy in feat: Add stream generation task scaffolding examples #3527
chore: bump version to 0.20.0rc0 by @ZhanruiSunCh in chore: bump version to 0.20.0rc0 #3561
chore: Add comments to modifications that fix TP size of DeepSeek-V3/R1 when using more than 16 GPUs by @jinyangyuan-nvidia in chore: Add comments to modifications that fix TP size of DeepSeek-V3/R1 when using more than 16 GPUs #3572
chore: waive test_llm_phi_quantization_1gpu by @QiJune in chore: waive test_llm_phi_quantization_1gpu #3603
feat: Support cos_sin_cache in all cases. by @yuxianq in feat: Support cos_sin_cache in all cases. #3517
fix: add SM90 guard for FP8 Blockscale GEMM by @lucifer1004 in fix: add SM90 guard for FP8 Blockscale GEMM #3575
infra: Update user list by @niukuo in infra: Update user list #3614
feat: Adding FP8 BMM from Codegen by @evezhier in feat: Adding FP8 BMM from Codegen #3541
waive test_llm_multi_node_with_postproc by @QiJune in waive test_llm_multi_node_with_postproc #3628
fix: Use hmac authentication for pickle encryption by @yibinl-nvidia in fix: Use hmac authentication for pickle encryption #3384
Clean up linear.py, mlp.py, gated_mlp.py by @hlu1 in Clean up linear.py, mlp.py, gated_mlp.py #3553
feat: Support CUDA graphs for EAGLE3 by @mikeiovine in feat: Support CUDA graphs for EAGLE3 #3176
feat: Nemotron-H model support by @vegaluisjose in feat: Nemotron-H model support #3430
waive test_fp8_scaled_mm by @QiJune in waive test_fp8_scaled_mm #3637
disable ib for ucx test by @chuangz0 in disable ib for ucx test #3613
tests: change qa perf test to trtllm-bench by @ruodil in tests: change qa perf test to trtllm-bench #3189
test: add quickstart test for nemotron-ultra by @crazydemo in test: add quickstart test for nemotron-ultra #3596
feat: Add support for smaller hidden_dim in AR fusion kernel by @yilin-void in feat: Add support for smaller hidden_dim in AR fusion kernel #3609
Fix rotary_emb param in NemotronH attention by @vegaluisjose in Fix rotary_emb param in NemotronH attention #3646
chore: Use ellipsis as default value to detect whether residual argument is provided by @yuxianq in chore: Use ellipsis as default value to detect whether residual argument is provided #3626
feat/loraOp by @danielafrimi in feat/loraOp #3455
Cherry-pick: update fp8 doc (update DeepSeek fp8 doc #3647) by @litaotju in Cherry-pick: update fp8 doc (#3647) #3650
test: [CI] Add failed cases into waives.txt by @xinhe-nv in test: [CI] Add failed cases into waives.txt #3627
Waive L0 tests by @yiqingy0 in Waive L0 tests #3651
feat: allocate minimal blocks per window size by @netanel-haber in feat: allocate minimal blocks per window size #3028
test: remove benchmark test list on main branch by @crazydemo in test: remove benchmark test list on main branch #3644
feat: Support unfused rope in MLA. by @yuxianq in feat: Support unfused rope in MLA. #3610
fix: Fix fused_moe cache fallback issue. by @hyukn in fix: Fix fused_moe cache fallback issue. #3652
fix: Correct reporting of text dtype for Llama 4. by @FrankD412 in fix: Correct reporting of text dtype for Llama 4. #3494
chore: waive test_llm_multi_node by @QiJune in chore: waive test_llm_multi_node #3664
chore: update multi gpu trigger file list by @QiJune in chore: update multi gpu trigger file list #3665
feat: adding multimodal (only image for now) support in trtllm-bench by @rakib-hasan in feat: adding multimodal (only image for now) support in trtllm-bench #3490
fix sage attention headsize check error in bertAttentionPlugin.cpp by @Jackch-NV in fix sage attention headsize check error in bertAttentionPlugin.cpp #3660
fix: llama4: address couple of issues in llama4 attention module by @chang-l in fix: llama4: address couple of issues in llama4 attention module #3491
chore: Refactor test_disaggregated.py by @Tabrizian in chore: Refactor test_disaggregated.py #3154
test: Add llama 4 to ci by @dongfengy in test: Add llama 4 to ci #3520
chore : Split more tests out of gpt tests by @peaceh-nv in chore : Split more tests out of gpt tests #3524
infra: Add step to generate new duration file by @EmmaQiaoCh in infra: Add step to generate new duration file #3298
refactor: Clean up CMakeLists.txt by @tongyuantongyu in refactor: Clean up CMakeLists.txt #3479
test: Unwaive test for nvbug_5150466 by @hchings in test: Unwaive test for nvbug_5150466 #3552
feat: Add Dynasor-CoT in scaffolding examples by @Fsanic in feat: Add Dynasor-CoT in scaffolding examples #3501
feat: Integrate GPUDirect Storage (GDS) into Executor API by @DomBrown in feat: Integrate GPUDirect Storage (GDS) into Executor API #3582
Remove dummy forward path by @HuiGao-NV in Remove dummy forward path #3669
fix: hmac in remote mpi session by @Superjomn in fix: hmac in remote mpi session #3649
test: add kv cache event tests for disagg workers by @zhengd-nv in test: add kv cache event tests for disagg workers #3602
chore: enable test_ptp_quickstart_advanced_mixed_precision back by @QiJune in chore: enable test_ptp_quickstart_advanced_mixed_precision back #3667
feat: Disaggregated router class by @pcastonguay in feat: Disaggregated router class #3584
Updating the run.py to make the draft target model run with the LLaMa 3 1B/8B by @mayani-nv in Updating the run.py to make the draft target model run with the LLaMa 3 1B/8B #3615
feat: trtllm-serve multimodal support by @yechank-nvidia in feat: trtllm-serve multimodal support #3590
chore: Waive disaggregated load balance by @Tabrizian in chore: Waive disaggregated load balance #3687
Clean up modeling_deepseek.py by @hlu1 in Clean up modeling_deepseek.py #3640
fix: Fix disaggregated load balance test by @Tabrizian in fix: Fix disaggregated load balance test #3689
feat: Introduce feature properties for attention backend. by @yuxianq in feat: Introduce feature properties for attention backend. #3659
test:update waives.txt for nvbug 5219532 by @nv-guomingz in test:update waives.txt for nvbug 5219532 #3672
test: Get Eagle tests working by @brb-nv in test: Get Eagle tests working #3593
move the reset models into examples/models/core directory by @QiJune in move the reset models into examples/models/core directory #3555
fix: Refactor Deepseek tp_size calculation by @hlu1 in fix: Refactor Deepseek tp_size calculation #3695
Update Nemotron Super and Ultra in Supported Models and add an example by @Naveassaf in Update Nemotron Super and Ultra in Supported Models and add an example #3632
infra: Add test list name check by @EmmaQiaoCh in infra: Add test list name check #3097
feat: [Deepseek] Add trtllm-gen MOE FP4 MOE backend by @hlu1 in feat: [Deepseek] Add trtllm-gen MOE FP4 MOE backend #3387
Waive L0 tests by @yiqingy0 in Waive L0 tests #3709
fix: update test_user_buffers_mm_add_prologue atol (fix: update test_user_buffers_mm_add_prologue atol #3711) by @liji-nv in fix: update test_user_buffers_mm_add_prologue atol (#3711) #3713
fix: Support TLLM_OVERRIDE_LAYER_NUM for llama4. by @yuxianq in fix: Support TLLM_OVERRIDE_LAYER_NUM for llama4. #3679
Report number of context tokens in one iteration by @HuiGao-NV in Report number of context tokens in one iteration #3691
fix: Remove ParallelConfig. by @yuxianq in fix: Remove ParallelConfig. #3678
feat: Offloading Multimodal embedding table to CPU in Chunked Prefill Mode by @katec846 in feat: Offloading Multimodal embedding table to CPU in Chunked Prefill Mode #3380
fix: fix cublas_scaled_mm by @dc3671 in fix: fix cublas_scaled_mm #3600
chore: update FMHA cubin files by @jinyangyuan-nvidia in chore: update FMHA cubin files #3680
test: add llama3.2 ptp test case by @StanleySun639 in test: add llama3.2 ptp test case #3363
bug: Fix hang bug when context server doesn't have enough capacity for KV Cache by @Tabrizian in bug: Fix hang bug when context server doesn't have enough capacity for KV Cache #3095
refact: use pybind block key and hasher in disagg worker test by @zhengd-nv in refact: use pybind block key and hasher in disagg worker test #3712
Fix: nvbugs/5232457 ModelOpt Mixtral AWQ OOM by @Barry-Delaney in Fix: nvbugs/5232457 ModelOpt Mixtral AWQ OOM #3714
ci: unwaive multi_node test by @Superjomn in ci: unwaive multi_node test #3715
Revert "Report number of context tokens in one iteration (Report number of context tokens in one iteration #3691)" by @kaiyux in Revert "Report number of context tokens in one iteration (#3691)" #3740
Fix/executor bugs by @byshiue in Fix/executor bugs #3681
test [TRTLLM-4477,TRTLLM-4481]: Accuracy test improvement (Part 3.5): Support GSM8K and GPQA by @syuoni in test [TRTLLM-4477,TRTLLM-4481]: Accuracy test improvement (Part 3.5): Support GSM8K and GPQA #3483
datasets API change : datasets.load_metric => evaluate.load by @rakib-hasan in datasets API change : datasets.load_metric => evaluate.load #3741
fix: Remove unnecessary max call by @kaiyux in fix: Remove unnecessary max call #3574
test: Unwaive Llama 3.1 with torch compile test by @yizhang-nv in test: Unwaive Llama 3.1 with torch compile test #3475
test: [CI] Add failed cases into waives.txt by @xinhe-nv in test: [CI] Add failed cases into waives.txt #3683
refactor: Introduce DecoderOutputBuffers per batch by @Funatiq in refactor: Introduce DecoderOutputBuffers per batch #3506
fix: fnmatch usage in modeling_utils.py (https://nvbugspro.nvidia.com/bug/5234567) by @syuoni in fix: fnmatch usage in modeling_utils.py (https://nvbugspro.nvidia.com/bug/5234567) #3754
test: change default max_batch_size to 512 in test config and dump config.json to log by @ruodil in test: change default max_batch_size to 512 in test config and dump config.json to log #3657
test: waive gemma on L20 by @crazydemo in test: waive gemma on L20 #3767
chore: remove useless allgather by @byshiue in chore: remove useless allgather #3751
feat: Unify two versions of allreduce custom op by @hyukn in feat: Unify two versions of allreduce custom op #3032
doc: Update doc for Deepseek min latency by @zongfeijing in doc: Update doc for Deepseek min latency #3717
Add log_level for disaggregated_mpi_worker by @qiaoxj07 in Add log_level for disaggregated_mpi_worker #3765
infra: Add test stages for sm120 by @EmmaQiaoCh in infra: Add test stages for sm120 #3533
feat: [AutoDeploy] generalizing cudagraph to multiple dynamic inputs by @lucaslie in feat: [AutoDeploy] generalizing cudagraph to multiple dynamic inputs #3589
chore: Move opencv-python import inside load_video() function by @AlessioNetti in chore: Move opencv-python import inside load_video() function #3768
Fixing the metric fmeasure access by @rakib-hasan in Fixing the metric fmeasure access #3774

New Contributors

@dcampora made their first contribution in chore: Mass integration of release/0.18 #3421
@narutolhy made their first contribution in feat: Add stream generation task scaffolding examples #3527
@evezhier made their first contribution in feat: Adding FP8 BMM from Codegen #3541
@vegaluisjose made their first contribution in feat: Nemotron-H model support #3430
@rakib-hasan made their first contribution in feat: adding multimodal (only image for now) support in trtllm-bench #3490
@Jackch-NV made their first contribution in fix sage attention headsize check error in bertAttentionPlugin.cpp #3660
@dongfengy made their first contribution in test: Add llama 4 to ci #3520
@Fsanic made their first contribution in feat: Add Dynasor-CoT in scaffolding examples #3501
@mayani-nv made their first contribution in Updating the run.py to make the draft target model run with the LLaMa 3 1B/8B #3615
@Naveassaf made their first contribution in Update Nemotron Super and Ultra in Supported Models and add an example #3632
@katec846 made their first contribution in feat: Offloading Multimodal embedding table to CPU in Chunked Prefill Mode #3380
@StanleySun639 made their first contribution in test: add llama3.2 ptp test case #3363
@Barry-Delaney made their first contribution in Fix: nvbugs/5232457 ModelOpt Mixtral AWQ OOM #3714
@yizhang-nv made their first contribution in test: Unwaive Llama 3.1 with torch compile test #3475
@qiaoxj07 made their first contribution in Add log_level for disaggregated_mpi_worker #3765
@AlessioNetti made their first contribution in chore: Move opencv-python import inside load_video() function #3768

Full Changelog: v0.19.0rc0...v0.20.0rc0

This discussion was created from the release v0.20.0rc0.

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment