Building the PaddlePaddle Inference C++ library from source for NVIDIA Blackwell
(RTX 50-series, compute capability 12.0 / sm_120) with CUDA 13 and TensorRT 10,
plus reproducible performance measurements for PP-OCRv5.
PaddlePaddle ships prebuilt Python wheels that include sm_120 kernels (the cu129
and cu130 package indexes). The prebuilt C++ inference library is a different story.
At the time of writing, the official Linux C++ download table tops out at:
| Backend | CUDA / cuDNN / TensorRT | gcc |
|---|---|---|
| CPU | — | 8.2 |
| GPU | CUDA 11.8 / cuDNN 8.9 / TRT 8.6 | 11.2 |
| GPU | CUDA 12.6 / cuDNN 9.5 / TRT 10.5 | 11.2 |
Blackwell support landed in CUDA 12.8. Both GPU builds above are compiled for
architectures ending at sm_90, so on an RTX 50-series card the predictor fails at
runtime:
The installed PaddlePaddle package was compiled for: 61 70 75 80 86 89 90
But your GPU is: 120
or, more generically:
CUDA error: no kernel image is available for execution on the device
If your project links libpaddle_inference.so from C++, there is no prebuilt library
that works. Building from source is the only path.
I could not find a public write-up covering this for the C++ library, so this repo documents the full process: the working CMake configuration, every obstacle hit along the way, and what the resulting performance actually looks like.
A single libpaddle_inference.so containing kernels for both sm_89 (Ada, RTX 40-series)
and sm_120 (Blackwell, RTX 50-series):
$ cuobjdump -lelf libpaddle_inference.so | grep -oE 'sm_[0-9]+' | sort -u
sm_120
sm_89$ cat version.txt
Paddle version: 3.3.0.dev20251226
GIT COMMIT ID: cbf3469113cd76b7d5f4cba7b8d7d5f55d9e9911
WITH_MKL: ON
WITH_ONEDNN: ON
WITH_GPU: ON
CUDA version: 13.0
CUDNN version: v9.16
WITH_TENSORRT: ON
TensorRT version: v10.14.1.48
CXX compiler version: 13.3.0One binary, both card generations, no rebuild when you swap GPUs.
| Component | Version |
|---|---|
| OS | Ubuntu 24.04.4 |
| GPU | RTX 5060 (compute capability 12.0) |
| Driver | 595.84 |
| CUDA | 13.0.88 |
| cuDNN | 9.16.0 |
| TensorRT | 10.14.1.48 (+cuda13.0 apt packages) |
| gcc/g++ | 13.3.0 |
| Paddle | v3.3.0 |
| RAM | 32 GB + 46 GB swap |
| CPU | 16 cores |
Other combinations may work. The two things that actually matter are a CUDA toolkit of 12.8 or newer and a TensorRT build matching your CUDA major version.
# 1. Toolchain sanity check — confirm nvcc emits sm_120
echo '__global__ void k(){} int main(){k<<<1,1>>>();return 0;}' > /tmp/t.cu
nvcc -arch=sm_120 /tmp/t.cu -o /tmp/t && echo TOOLCHAIN_OK
# 2. Source
git clone https://github.com/PaddlePaddle/Paddle.git
cd Paddle && git checkout v3.3.0
git submodule sync --recursive && git submodule update --init --recursive
# 3. Protect the build from the OOM killer (see docs/TROUBLESHOOTING.md)
sudo systemctl mask systemd-oomd.service systemd-oomd.socket
sudo systemctl stop systemd-oomd.socket systemd-oomd.service
sudo sysctl -w vm.overcommit_memory=1
# 4. Configure + build
PADDLE_SRC=$PWD bash /path/to/scripts/configure.sh
PADDLE_SRC=$PWD bash /path/to/scripts/build.sh
# 5. Verify sm_120 is actually in the binary
cuobjdump -lelf build/paddle_inference_install_dir/paddle/lib/libpaddle_inference.so \
| grep -oE 'sm_[0-9]+' | sort -uExpect 3–5 hours for a clean build.
Full walkthrough with verification checkpoints: docs/BUILD.md
-DCUDA_ARCH_NAME=Manual
-DCUDA_ARCH_BIN="89;120" # both architectures in one binary
-DON_INFER=ON # produces paddle_inference_install_dir
-DWITH_PYTHON=OFF # C++ only
-DWITH_NCCL=OFF # REQUIRED — see below
-DWITH_TENSORRT=ON -DTENSORRT_ROOT=/usrTwo notes worth their own line:
WITH_NCCL=OFFis not optional for a single-GPU build.WITH_DISTRIBUTE=OFFalone does not disable the NCCL header check, and configure fails onNCCL_INCLUDE_DIR-NOTFOUND/nccl.h.CUDA_ARCH_NAME=Manualmust come beforeCUDA_ARCH_BINor the arch list is ignored. Confirm it took effect in the configure output:-- NVCC_FLAGS_EXTRA: -gencode arch=compute_89,code=sm_89 -gencode arch=compute_120,code=sm_120
For apt-installed TensorRT, TENSORRT_ROOT=/usr — headers live in
/usr/include/x86_64-linux-gnu, libraries in /usr/lib/x86_64-linux-gnu.
There is no TensorRT-10.x/ directory.
Seven things went wrong. Each is documented with symptom, cause and fix in docs/TROUBLESHOOTING.md:
| # | Symptom | Cause |
|---|---|---|
| 1 | NCCL_INCLUDE_DIR-NOTFOUND/nccl.h |
WITH_DISTRIBUTE=OFF doesn't disable NCCL |
| 2 | Killed mid-build, cicc OOM at 2 GB RSS |
systemd-oomd kills preemptively |
| 3 | Build dies silently after hours | tmux session never actually started |
| 4 | -DWITH_FLASHATTN=OFF has no effect |
flag doesn't exist in v3.3.0 |
| 5 | libiomp5.so not found at runtime |
mklml/onednn dirs missing from rpath |
| 6 | libabsl_*.so not found |
third-party build dirs missing from rpath |
| 7 | please compile with TENSORRT first |
app hardcodes EnableTensorRtEngine |
Number 2 is the one that will cost you the most time. systemd-oomd on Ubuntu 24.04
kills processes based on memory pressure, not exhaustion — it killed cicc while
28 GB of RAM and 46 GB of swap were free.
Measured with PP-OCRv5_mobile (detection + recognition) on the RTX 5060. Every number is the mean of 50 iterations after 5 warmup runs.
| Input | Dimensions | Text lines | Time |
|---|---|---|---|
| single line crop | 594 × 80 | 1 | 5.9 ms |
| same, stretched | 594 × 245 | 1 | 9.1 ms |
| full frame | 594 × 245 | 3 | 29.9 ms |
| larger frame | 1049 × 311 | 3 | 33.2 ms |
Runtime scales with the number of detected text lines, not with image size. Roughly 10 ms per additional line.
Twelve optimisation attempts. Eleven changed nothing:
| Attempt | Result |
|---|---|
| TensorRT enabled | no change |
| FP16 precision | no change |
| CUDA graphs | no change |
| PIR (new IR) disabled | no change |
| Batching 1 → 16 images | perfectly linear (no gain) |
max_candidates 1000 → 50 |
no change |
resize_long 960 → 640 |
no change |
| Recognition dictionary 18,387 → 40 chars | no change |
Recognition batch_size 32 → 1 |
no change |
| Dead IoU computation removed | no change |
INTER_CUBIC → INTER_LINEAR in crop |
no change |
| Cropping to one line instead of three | 5× faster |
warpPerspective was instrumented directly and averages 0.167 ms per crop — the
crop stage is 1.7 % of total runtime, so moving preprocessing to the GPU is not worth
the effort.
Method and raw numbers: docs/BENCHMARKS.md
If your pipeline already localises the region of interest (an object detector, a fixed camera position), crop to the specific line you need rather than the whole text block. That is a 5× speedup for free, and it dwarfs anything achievable through inference-engine tuning.
For a 20 fps camera (50 ms per frame budget), one line of OCR costs 5.9 ms and leaves 44 ms for everything else.
| Script | Purpose |
|---|---|
scripts/preflight.sh |
Verify toolchain, driver, RAM, disk before starting |
scripts/configure.sh |
CMake configuration with the flags above |
scripts/build.sh |
Build inference_lib_dist |
scripts/watch_build.sh |
Progress + memory monitor, detects silent death |
scripts/verify.sh |
Confirm sm_120 is in the binary |
All read PADDLE_SRC and CUDA_ARCH from the environment:
export PADDLE_SRC=$HOME/Paddle
export CUDA_ARCH="89;120" # or just "120" for Blackwell only
bash scripts/configure.shNo prebuilt binaries. The resulting library is ~720 MB and tightly coupled to the exact CUDA, cuDNN, TensorRT and gcc versions on the build machine. Distributing it would mean shipping an unofficial, unsupported build of someone else's Apache-2.0 project. Build it yourself with the scripts here — you get a library matched to your environment, and you know what went into it.
This repository contains documentation, build scripts and benchmark data. It is released under the MIT licence.
PaddlePaddle is a separate project by Baidu, licensed under Apache 2.0. Nothing here modifies Paddle source code — these are build configurations and measurements. Any library you produce with these scripts is a build of Paddle and remains subject to Paddle's licence and terms.
This is an unofficial guide with no affiliation to Baidu or the PaddlePaddle team.
Mahmoud Shehata Ahmed
Corrections, additions and reports from other Blackwell cards are welcome — open an issue.