[chenfeiz0326]

Description

examples/test_gpt.py::test_gpt_oss_20b_lora_torch[gpt-oss-20b-lora-adapter_NIM_r8-gpt-oss-20b] fails on H100 (L0_PostMerge 3021, DGX_H100-PyTorch-Post-Merge-1) with CUDA error: an illegal memory access was encountered at the first CUDA-graph LoRA decode step.

Root cause. loraGroupGEMMParamFillRowReorderFusionKernel runs with at least 256 threads per block. In the OUT_SIZES_INDEX block, every thread reads module_out_sizes[module_id], where module_id = threadIdx.x / max_lora_count. That tensor has only MODULE_COUNT entries (3 int32 = 12 bytes for fused QKV), so threads past max_lora_count * MODULE_COUNT read out of bounds. BlockStore3 drops their rows, so the result was never wrong, but the read faults whenever the allocation sits at the end of mapped memory. A GPU coredump names this kernel and block (Warp MMU Fault on the module_out_sizes load). With the caching allocator disabled, compute-sanitizer reports 10 invalid 4-byte reads past the 12-byte allocation. #20188 changed the allocation layout enough that this test now hits it every time; the bug itself dates back to #8279.

Fix. Read module_out_sizes only for module_id < MODULE_COUNT. The output for real problems is unchanged.

Test Coverage

On computelab H100-80GB with the CI image pytorch-26.09-...-202610130652-20175:

• Same source built with and without the fix, failing test run 5x each: base fails 5/5 with the IMA, fix passes 5/5 with the expected text.
• The same op call under compute-sanitizer memcheck (PYTORCH_NO_CUDA_MEMORY_CACHING=1): 10 errors before, 0 after; out_sizes is correct.
• tests/unittest/_torch/peft/{test_lora,test_lora_autotuner,test_fp8_lora_grouped_gemm_regressions,test_moe_lora_cuda_graph_params}.py and thop/parallel_hw_agnostic/test_custom_ops.py: 37 passed.
• l0_h100 LoRA entries unittest/_torch/modules/tests_lora_modules and test_qwen_moe_routed_expert_multi_lora_varying_ranks[eager|cudagraph]: passed.

No waiver exists for this bug.

Fixes https://nvbugs/6939562

🤖 Generated with Claude Code

Dev Engineer Review

• The kernel now checks module_id < MODULE_COUNT before reading module_out_sizes. This prevents out-of-range reads from threads beyond the valid module range.
• The output-size row remains zero-initialized unless b_ptr is non-null and the module ID is valid. The change does not alter the public API.
• The reported CUDA-graph failure and test results were not independently verified.

QA Engineer Review

No test changes.

Per-File QA Perspective

• cpp/tensorrt_llm/kernels/lora/loraGroupGEMMParamFillRowReorderFusion.cu: Verify output-size rows for valid modules and null b_ptr values, plus threads beyond MODULE_COUNT. Confirm CUDA-graph LoRA decode no longer triggers invalid reads.

[coderabbitai]

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 4b3e15b9-c0c2-494f-9bae-d380708ff022

📥 Commits

Reviewing files that changed from the base of the PR and between 739cdb2 and df4e531.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: fae5bb83-f0b3-4fdb-b06b-25ac1048b3e9

📥 Commits

Reviewing files that changed from the base of the PR and between dac84b0 and 739cdb2.

📒 Files selected for processing: 1

• cpp/tensorrt_llm/kernels/lora/loraGroupGEMMParamFillRowReorderFusion.cu

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The output-sizes kernel block now checks the B pointer and module ID before populating a row. Invalid or inactive rows retain their zero-initialized values. The copyright notice now covers 2025–2026.

Changes

LoRA output-size row population

Layer / File(s)	Summary
Guard output-size row population cpp/tensorrt_llm/kernels/lora/loraGroupGEMMParamFillRowReorderFusion.cu	The kernel populates output-size rows only when the B pointer is nonzero and the module ID is valid. Otherwise, it stores the zero-initialized row. The copyright notice now covers 2025–2026.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~5 minutes

Change: Bug fix

Suggested reviewers: brnguyen2, jychang012

Merge Risk: ⚪ Minimal · up to 739cd

The bounds check addresses the reported invalid-read risk, and no remaining merge-blocking issue was identified.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly identifies the NVBugs issue, fix type, and primary change: bounding the module_out_sizes read in the LoRA parameter-fill kernel.
Description check		The description explains the failure, root cause, fix, and extensive test coverage. It does not explicitly complete the PR Checklist, but the required technical information is otherwise present.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[chenfeiz0326]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #78706 [ run ] triggered by Bot. Commit: 739cdb2 Link to invocation

[chenfeiz0326]

@NVIDIA/trt-llm-kernels-devs: requesting a code-owner review. This is a one-hunk fix for NVBug 6939562 (post-merge test_gpt_oss_20b_lora_torch illegal memory access on H100).

loraGroupGEMMParamFillRowReorderFusionKernel's OUT_SIZES block runs 256+ threads, and every thread read module_out_sizes[module_id] even though the tensor has only MODULE_COUNT entries. The fix keeps that read inside module_id < MODULE_COUNT; output for real (lora, module) problems is unchanged. compute-sanitizer shows 10 OOB reads before and 0 after. The failing test fails 5/5 without the fix and passes 5/5 with it on H100. Pre-merge CI: PR_Github #78706 (running).

[chenfeiz0326]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #78706 [ run ] completed with state SUCCESS. Commit: 739cdb2
/LLM/main/L0_MergeRequest_PR pipeline #64848 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[tensorrt-cicd]

PR_Github #78713 [ run ] triggered by Bot. Commit: 739cdb2 Link to invocation

[tensorrt-cicd]

PR_Github #78713 [ run ] completed with state SUCCESS. Commit: 739cdb2
/LLM/main/L0_MergeRequest_PR pipeline #64853 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chenfeiz0326]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #78729 [ run ] triggered by Bot. Commit: 739cdb2 Link to invocation

[tensorrt-cicd]

PR_Github #78729 [ run ] completed with state FAILURE. Commit: 739cdb2
/LLM/main/L0_MergeRequest_PR pipeline #64867 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chenfeiz0326]

Pre-merge CI on head 739cdb2 has failed three times in a row, each time only on stage H100_PCIe-PyTorch-Perf-1, and each failure is unrelated to this PR (the PR changes one bounds check in the LoRA kernel loraGroupGEMMParamFillRowReorderFusion.cu):

Run	Failure in H100_PCIe-PyTorch-Perf-1
L0_MergeRequest_PR #64848	test_perf[llama_v3.1_8b_instruct-bench-pytorch-float16-input_output_len:128,128-reqs:8192]: CUDA-capable device(s) is/are busy or unavailable (cuDevicePrimaryCtxRetain 999) on host ipp1-3152. The B300-PyTorch-1 KVCM2 CorruptedError/CuOOMError failures in the same run also appear in other PRs' build 64822.
L0_MergeRequest_PR #64853	same test: hang watchdog fired, Test terminated unexpectedly (host ipp2-2359); the other 5110 results passed
L0_MergeRequest_PR #64867	stage aborted before any test ran: pod h100-cr---tensorrt-github2-llm-main-l0-test-9579-d5ff67-j-b8ngf, Container trt-llm was terminated (Exit Code: 128, Reason: StartError); all other downstream jobs SUCCESS

That perf test passed in other PRs' builds 64839, 64840 and 64841. The LoRA tests that ran in #64848 passed.

Could someone from CI infra look at the H100_PCIe perf pool, and could a code owner (@NVIDIA/trt-llm-kernels-devs) review the change?

[chenfeiz0326]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #78737 [ run ] triggered by Bot. Commit: 739cdb2 Link to invocation

[tensorrt-cicd]

PR_Github #78737 [ run ] completed with state SUCCESS. Commit: 739cdb2
/LLM/main/L0_MergeRequest_PR pipeline #64875 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chenfeiz0326]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #78832 [ run ] triggered by Bot. Commit: df4e531 Link to invocation

[chenfeiz0326]

@NVIDIA/trt-llm-kernels-devs: following up on the review request. This PR is a single bounds check in loraGroupGEMMParamFillRowReorderFusion.cu: the OUT_SIZES block now reads module_out_sizes[module_id] only when module_id < MODULE_COUNT. It fixes NVBug 6939562: test_gpt_oss_20b_lora_torch failed with an illegal memory access in L0_PostMerge DGX_H100 #3021 through #3026. #3027 passed, but main still has the out-of-bounds read, and whether it faults depends on memory layout. The earlier pre-merge runs failed only in stages unrelated to this change: B300-PyTorch-1 KVCM2/OOM errors in the first run (also seen on other PRs), and H100_PCIe-PyTorch-Perf-1 infra failures in the later runs (device unavailable, a hang, a pod start error, a killed benchmark). I've rebased onto main, and pre-merge CI is re-running on df4e531. A code-owner approval is the only other thing blocking merge.

[github-actions]

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

[tensorrt-cicd]

PR_Github #78832 [ run ] completed with state SUCCESS. Commit: df4e531
/LLM/main/L0_MergeRequest_PR pipeline #64951 completed with status: 'SUCCESS'

CI Report

Link to invocation