Repository navigation
Add onnxruntime-gpu (ONNX-on-GPU) install steps to TensorRT doc - #12
Merged
Merged
Conversation
added 2 commits
May 31, 2026 14:14
New section in docs/TensorRT.md covering how to run the ONNX Deploy path on GPU via onnxruntime-gpu: CUDA/cuDNN version matching (1.17.x->CUDA 11.8/cuDNN 8; 1.18+->CUDA 12/cuDNN 9), CUDA 11.8 vs 12.x install commands, PATH notes, and verification. Troubleshooting renumbered to section 9.
Make explicit that onnxruntime-gpu does not bundle CUDA/cuDNN (it loads them from PATH), that PyTorch's bundled libs don't satisfy it, and that a working setup is the right build PLUS matching CUDA/cuDNN on PATH. Reframe as Route A (match CUDA 12, simplest with a recent onnxruntime) vs Route B (match CUDA 11.8 + cuDNN 8), and clarify the cuDNN major pairs with the onnxruntime build.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a section to docs/TensorRT.md on running the plain ONNX Deploy path on GPU via
onnxruntime-gpu(no TensorRT needed):onnxruntime-gpu 1.17.x→ CUDA 11.8 / cuDNN 8;1.18+→ CUDA 12 / cuDNN 9.cu118) and CUDA 12.x cases, with PATH/cuDNN notes and the "only one of onnxruntime/onnxruntime-gpu" caveat.get_available_providers()→CUDAExecutionProvider; Deploy side panel shows the provider).Directly addresses the earlier "ONNX is slower than .pth" finding (it was falling back to CPU because onnxruntime-gpu didn't match the installed CUDA). Troubleshooting renumbered to §9. Docs-only.