[HW Accel Support]: Failed to load library libonnxruntime_providers_cuda.so with error: libcufft.so.10: cannot open shared object file: No such file or directory

[SmithAdamL]

Describe the problem you are having

I have been trying to dig in to this for days now, to no luck. I was successfully utilizing my NVIDIA RTX 3060 Ti to run Utilizing the nvidia tensorrt detector for over a month no problem. I switched to Frigate Plus and have to make the change to ONNX to support the custom models. Ever since then, I have had nothing but problems. I can get everything working, except the HW Acceleration. It fails with this error every time. I have done a full clean of the Cuda drivers, and reinstalled multiple times. I have validated the card is available and visible in the VM and the Docker Container - and confirmed everything is running. Some Reason - this error persists!

Version

0.15.0-cea210d

Frigate config file

version: 0.15-1

mqtt:
  enabled: true
  host: 10.10.10.4        # Adjust if your MQTT broker is hosted elsewhere.
  port: 1883
  user: MQTTUSER
  password: 'XXXXXX'
  topic_prefix: frigate
  client_id: frigate
  stats_interval: 60

auth:
  enabled: false

ffmpeg:
  hwaccel_args: preset-nvidia

objects:
  track:
    - person
    - face
    - care
    - motorcycle
    - bicycle
    - boat
    - license_plate
    - amazon
    - usps
    - ups
    - fedex
    - dog
    - cat
    - deer
    - bird
    - raccoon
    - fox
    - cow
    - squirrel
    - goat
    - rabbit
    - package

snapshots:
  enabled: true
  clean_copy: true
  timestamp: true
  bounding_box: true
  crop: false
  retain:
    default: 7
    objects:
      person: 320
      car: 30
      motorcycle: 30
      bicycle: 30

go2rtc: {}

cameras:
  566_Back_Door:
    ffmpeg:
      inputs:
        - path: rtsp://go2rtc:8556/566_Back_Door_detect
          roles:
            - detect
        - path: rtsp://go2rtc:8556/566_Back_Door
          roles:
            - record
    detect:
      width: 640
      height: 480
      fps: 7

  566_Woods_Trails1:
    ffmpeg:
      inputs:
        - path: rtsp://go2rtc:8556/566_Woods_Trails1_detect
          roles:
            - detect
        - path: rtsp://go2rtc:8556/566_Woods_Trails1
          roles:
            - record
    detect:
      width: 640
      height: 480
      fps: 7

  566_Front_PTZ:
    ffmpeg:
      inputs:
        - path: rtsp://go2rtc:8556/566_Front_PTZ_detect
          roles:
            - detect
        - path: rtsp://go2rtc:8556/566_Front_PTZ
          roles:
            - record
    detect:
      width: 640
      height: 480
      fps: 7

  566_Woods_Back_Yard:
    ffmpeg:
      inputs:
        - path: rtsp://go2rtc:8556/566_Woods_Back_Yard_detect
          roles:
            - detect
        - path: rtsp://go2rtc:8556/566_Woods_Back_Yard
          roles:
            - record
    detect:
      width: 640
      height: 480
      fps: 5

  95_Drive_Entry:
    ffmpeg:
      inputs:
        - path: rtsp://go2rtc:8556/95_Drive_Entry_detect
          roles:
            - detect
        - path: rtsp://go2rtc:8556/95_Drive_Entry
          roles:
            - record
    detect:
      width: 640
      height: 480
      fps: 5

  566_Carport:
    ffmpeg:
      inputs:
        - path: rtsp://go2rtc:8556/566_Carport_detect
          roles:
            - detect
        - path: rtsp://go2rtc:8556/566_Carport
          roles:
            - record
    detect:
      width: 640
      height: 480
      fps: 7

  566_Shop:
    ffmpeg:
      inputs:
        - path: rtsp://go2rtc:8556/566_Shop_detect
          roles:
            - detect
        - path: rtsp://go2rtc:8556/566_Shop
          roles:
            - record
    detect:
      width: 640
      height: 480
      fps: 5


detectors:
  onnx:
    type: onnx
    device: "0"

model:
  path: plus://XXXXXX
  model_type: yolonas
  input_tensor: nchw
  input_pixel_format: rgb
  width: 320
  height: 320

record:
  enabled: true
  retain:
    days: 0        # Disable continuous (24/7) recording.
    mode: all
  alerts:
    pre_capture: 10
    post_capture: 60
    retain:
      days: 2
      mode: active_objects

docker-compose file or Docker CLI command

services:
  mosquitto:
    image: eclipse-mosquitto:latest
    container_name: mosquitto
    restart: unless-stopped
    ports:
      - "1883:1883"
      - "9001:9001"
    volumes:
      - ./mosquitto/config:/mosquitto/config
      - ./mosquitto/data:/mosquitto/data
      - ./mosquitto/log:/mosquitto/log
      - /etc/localtime:/etc/localtime:ro

  frigate:
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
    container_name: frigate
    restart: unless-stopped
    gpus: all
    privileged: true
    shm_size: "6144mb"
    ports:
      - "5000:5000"        # Frigate Web UI and API
      - "8554:8554"        # RTSP restreaming
      - "8555:8555/tcp"    # WebRTC TCP
      - "8555:8555/udp"    # WebRTC UDP
    environment:
      FRIGATE_RTSP_PASSWORD: "XXXXX"
      NVIDIA_VISIBLE_DEVICES: "all"
      NVIDIA_DRIVER_CAPABILITIES: "compute,utility,video"
      LD_LIBRARY_PATH: "/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH"
      YOLO_MODELS: yolov7-320,yolov7x-640
      TRT_MODEL_PREP_DEVICE: 0
      PLUS_API_KEY: XXXXXX
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    volumes:
      - ./frigate/config:/config
      - ./frigate/media:/media/frigate
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 1073741824  # 1GB in bytes

  go2rtc:
    image: alexxit/go2rtc:latest
    container_name: go2rtc
    restart: unless-stopped
    ports:
      - "3084:3084"        # Web UI and API
      - "8556:8556"        # RTSP streaming
      - "9555:9555/udp"    # WebRTC UDP
      - "9555:9555/tcp"    # WebRTC TCP
      - "8086:8086"        # HTTP MJPEG/HLS Streaming
    volumes:
      - ./go2rtc/config:/config

Relevant Frigate log output

s6-rc: info: service s6rc-fdholder: starting
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service s6rc-fdholder successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service trt-model-prepare: starting
s6-rc: info: service log-prepare: starting
/etc/s6-overlay/s6-rc.d/trt-model-prepare/run: line 23: [: missing `]'
s6-rc: info: service log-prepare successfully started
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service certsync-log: starting
s6-rc: info: service nginx-log successfully started
s6-rc: info: service go2rtc-log successfully started
No models to convert.
s6-rc: info: service go2rtc: starting
s6-rc: info: service trt-model-prepare successfully started
s6-rc: info: service frigate-log successfully started
s6-rc: info: service certsync-log successfully started
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service frigate: starting
s6-rc: info: service go2rtc-healthcheck successfully started
s6-rc: info: service frigate successfully started
s6-rc: info: service nginx: starting
2025-03-05 03:33:08.066313126  [INFO] Preparing new go2rtc config...
2025-03-05 03:33:08.066925900  [INFO] Starting NGINX...
2025-03-05 03:33:08.067094029  [INFO] Preparing Frigate...
2025-03-05 03:33:08.077409316  [INFO] No TLS certificate found. Generating a self signed certificate...
2025-03-05 03:33:08.189063574  [INFO] Starting Frigate...
2025-03-05 03:33:08.575627759  [INFO] Starting go2rtc...
2025-03-05 03:33:08.644328577  03:33:08.644 INF go2rtc platform=linux/amd64 revision=b2399f3 version=1.9.2
2025-03-05 03:33:08.644331685  03:33:08.644 INF config path=/dev/shm/go2rtc.yaml
2025-03-05 03:33:08.644620796  03:33:08.644 INF [rtsp] listen addr=:8554
2025-03-05 03:33:08.644644471  03:33:08.644 INF [api] listen addr=:1984
2025-03-05 03:33:08.644778921  03:33:08.644 INF [webrtc] listen addr=:8555/tcp
s6-rc: info: service nginx successfully started
s6-rc: info: service certsync: starting
s6-rc: info: service certsync successfully started
s6-rc: info: service legacy-services: starting
2025-03-05 03:33:09.557343517  [INFO] Starting certsync...
s6-rc: info: service legacy-services successfully started
2025-03-05 03:33:09.615132328  127.0.0.1 - - [05/Mar/2025:03:33:09 +0000] "" 400 0 "-" "-" "-"
2025-03-05 03:33:10.779102383  [2025-03-05 03:33:10] frigate.util.config            INFO    : Checking if frigate config needs migration...
2025-03-05 03:33:10.797884619  [2025-03-05 03:33:10] frigate.util.config            INFO    : frigate config does not need migration...
2025-03-05 03:33:10.901690591  [2025-03-05 03:33:10] frigate.app                    INFO    : Starting Frigate (0.15.0-cea210d)
2025-03-05 03:33:10.918970766  [2025-03-05 03:33:10] peewee_migrate.logs            INFO    : Starting migrations
2025-03-05 03:33:10.919210774  [2025-03-05 03:33:10] peewee_migrate.logs            INFO    : There is nothing to migrate
2025-03-05 03:33:10.930081550  [2025-03-05 03:33:10] frigate.app                    INFO    : Recording process started: 361
2025-03-05 03:33:10.930355058  [2025-03-05 03:33:10] frigate.app                    INFO    : Review process started: 362
2025-03-05 03:33:10.932082781  [2025-03-05 03:33:10] frigate.app                    INFO    : go2rtc process pid: 107
2025-03-05 03:33:10.941214537  [2025-03-05 03:33:10] detector.onnx                  INFO    : Starting detection process: 377
2025-03-05 03:33:10.972380108  [2025-03-05 03:33:10] frigate.app                    INFO    : Output process started: 398
2025-03-05 03:33:10.982943226  [2025-03-05 03:33:10] frigate.detectors.plugins.onnx INFO    : ONNX: loaded onnxruntime module
2025-03-05 03:33:10.982945593  [2025-03-05 03:33:10] frigate.detectors.plugins.onnx INFO    : ONNX: loading /config/model_cache/ca05f208e91ebc8b1c6a20adeb86ddc5
2025-03-05 03:33:10.988469351  2025-03-05 03:33:10.988439331 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcufft.so.10: cannot open shared object file: No such file or directory
2025-03-05 03:33:10.988471400
2025-03-05 03:33:10.988498465  2025-03-05 03:33:10.988490372 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:895 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirementsto ensure all dependencies are met.
2025-03-05 03:33:11.091486905  [2025-03-05 03:33:11] frigate.detectors.plugins.onnx INFO    : ONNX: /config/model_cache/ca05f208e91ebc8b1c6a20adeb86ddc5 loaded
2025-03-05 03:33:11.430782099  [2025-03-05 03:33:11] frigate.app                    INFO    : Camera processor started for 566_Back_Door: 468
2025-03-05 03:33:11.437465570  [2025-03-05 03:33:11] frigate.app                    INFO    : Camera processor started for 566_Woods_Trails1: 469
2025-03-05 03:33:11.453725876  [2025-03-05 03:33:11] frigate.app                    INFO    : Camera processor started for 566_Front_PTZ: 477
2025-03-05 03:33:11.453964417  [2025-03-05 03:33:11] frigate.app                    INFO    : Camera processor started for 566_Woods_Back_Yard: 490
2025-03-05 03:33:11.458981269  [2025-03-05 03:33:11] frigate.app                    INFO    : Camera processor started for 95_Drive_Entry: 495
2025-03-05 03:33:11.459064495  [2025-03-05 03:33:11] frigate.app                    INFO    : Camera processor started for 566_Carport: 506
2025-03-05 03:33:11.459740417  [2025-03-05 03:33:11] frigate.app                    INFO    : Camera processor started for 566_Shop: 511
2025-03-05 03:33:11.479227713  [2025-03-05 03:33:11] frigate.app                    INFO    : Capture process started for 566_Back_Door: 524
2025-03-05 03:33:11.490320775  [2025-03-05 03:33:11] frigate.app                    INFO    : Capture process started for 566_Woods_Trails1: 527
2025-03-05 03:33:11.490372610  [2025-03-05 03:33:11] frigate.app                    INFO    : Capture process started for 566_Front_PTZ: 531
2025-03-05 03:33:11.504428541  [2025-03-05 03:33:11] frigate.app                    INFO    : Capture process started for 566_Woods_Back_Yard: 542
2025-03-05 03:33:11.510668110  [2025-03-05 03:33:11] frigate.app                    INFO    : Capture process started for 95_Drive_Entry: 547
2025-03-05 03:33:11.522755172  [2025-03-05 03:33:11] frigate.app                    INFO    : Capture process started for 566_Carport: 554
2025-03-05 03:33:11.525434861  [2025-03-05 03:33:11] frigate.app                    INFO    : Capture process started for 566_Shop: 559
2025-03-05 03:33:11.669534661  [2025-03-05 03:33:11] frigate.api.fastapi_app        INFO    : Starting FastAPI app
2025-03-05 03:33:11.703544239  2025/03/05 03:33:11 [error] 179#179: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.10.2, server: , request: "GET /api/stats HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "10.10.10.4:5000"
2025-03-05 03:33:11.703547460  2025/03/05 03:33:11 [error] 179#179: *2 auth request unexpected status: 502 while sending to client, client: 10.10.10.2, server: , request: "GET /api/stats HTTP/1.1", host: "10.10.10.4:5000"
2025-03-05 03:33:11.750718010  [2025-03-05 03:33:11] frigate.api.fastapi_app        INFO    : FastAPI started
2025-03-05 03:33:18.073476440  [INFO] Starting go2rtc healthcheck service...
2025-03-05 03:34:09.647969134  127.0.0.1 - - [05/Mar/2025:03:34:09 +0000] "" 400 0 "-" "-" "-"
2025-03-05 03:35:09.682970629  127.0.0.1 - - [05/Mar/2025:03:35:09 +0000] "" 400 0 "-" "-" "-"

Relevant go2rtc log output

Type
Timestamp
Tag
Message
info
2025-03-05 03:33:08
startup
Preparing new go2rtc config...
info
2025-03-05 03:33:08
startup
Starting go2rtc...
info
2025-03-05 03:33:08
startup
go2rtc platform=linux/amd64 revision=b2399f3 version=1.9.2
info
2025-03-05 03:33:08
startup
config path=/dev/shm/go2rtc.yaml
info
2025-03-05 03:33:08
rtsp
listen addr=:8554
info
2025-03-05 03:33:08
api
listen addr=:1984
info
2025-03-05 03:33:08
webrtc
listen addr=:8555/tcp
info
2025-03-05 03:33:18
startup
Starting go2rtc healthcheck service...

FFprobe output from your camera

ffprobe version 6.1.1-3ubuntu5 Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 13 (Ubuntu 13.2.0-23ubuntu3)
  configuration: --prefix=/usr --extra-version=3ubuntu5 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --disable-omx --enable-gnutls --enable-libaom --enable-libass --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libharfbuzz --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-openal --enable-opencl --enable-opengl --disable-sndio --enable-libvpl --disable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-ladspa --enable-libbluray --enable-libjack --enable-libpulse --enable-librabbitmq --enable-librist --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libx264 --enable-libzmq --enable-libzvbi --enable-lv2 --enable-sdl2 --enable-libplacebo --enable-librav1e --enable-pocketsphinx --enable-librsvg --enable-libjxl --enable-shared
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
[rtsp @ 0x5df3737a6f80] max delay reached. need to consume packet
[rtsp @ 0x5df3737a6f80] RTP: missed 154 packets
[rtsp @ 0x5df3737a6f80] max delay reached. need to consume packet
[rtsp @ 0x5df3737a6f80] RTP: missed 46 packets
[h264 @ 0x5df3737ab640] concealing 18371 DC, 18371 AC, 18371 MV errors in I frame
[rtsp @ 0x5df3737a6f80] max delay reached. need to consume packet
[rtsp @ 0x5df3737a6f80] RTP: missed 4 packets
[h264 @ 0x5df3737ab640] concealing 3565 DC, 3565 AC, 3565 MV errors in P frame
[rtsp @ 0x5df3737a6f80] max delay reached. need to consume packet
[rtsp @ 0x5df3737a6f80] RTP: missed 1 packets
Input #0, rtsp, from 'rtsp://XXX:[email protected]/h264Preview_01_main':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h264Preview_01_main
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 3072x1728, 23.75 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp

Install method

Proxmox via Docker

Object Detector

TensorRT

Network connection

Wired

Camera make and model

Amcrest, Reolink, SV3C

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

[NickM-27]

The model config is wrong, only the path should be set to plus everything else should not be set.

And I'd suggest remove the ld_library_path override then run again and show new logs