SFM Stage Error: cudaMemcpyToSymbol failed

[cturner8725]

Hello All,

I am running into an error with gpu proccessing during the SFM stage with gives me the following message

“cudaMemcpyToSymbol failed for Gauss kernel initialization: unrecognized error code”

I am using the Windows Native WebODM version 2.7.0 on Windows 11 24H2 with all the latest drivers. GPU is a RTX 5080. When I restart the task, the SFM stage will proceed using my CPU. When the MVS stage begins, it will process on my GPU without error. This has been replicated multiple times with different data sets.

Can someone help me find a solution to this?

Photos and Tasks Output Exports: https://drive.google.com/drive/folders/1TgJm_hz7dHh6OIXDHHVVz6A11FnGtoQc?usp=drive_link

Parameters: Default; Resize Images 2048

[INFO] Initializing ODM 3.5.4 - Fri Feb 21 19:03:40 2025
[INFO] ==============
[INFO] 3d_tiles: False
[INFO] align: None
[INFO] auto_boundary: True
[INFO] auto_boundary_distance: 0
[INFO] bg_removal: False
[INFO] boundary: {}
[INFO] build_overviews: False
[INFO] camera_lens: auto
[INFO] cameras: {}
[INFO] cog: True
[INFO] copy_to: None
[INFO] crop: 3
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: False
[INFO] dem_gapfill_steps: 3
[INFO] dem_resolution: 5
[INFO] dsm: True
[INFO] dtm: False
[INFO] end_with: odm_postprocess
[INFO] fast_orthophoto: False
[INFO] feature_quality: high
[INFO] feature_type: dspsift
[INFO] force_gps: False
[INFO] gcp: None
[INFO] geo: None
[INFO] gltf: True
[INFO] gps_accuracy: 3
[INFO] ignore_gsd: False
[INFO] matcher_neighbors: 0
[INFO] matcher_order: 0
[INFO] matcher_type: flann
[INFO] max_concurrency: 16
[INFO] merge: all
[INFO] mesh_octree_depth: 11
[INFO] mesh_size: 200000
[INFO] min_num_features: 10000
[INFO] name: 66102c90-90ea-458a-ac16-388a2606ae2a
[INFO] no_gpu: False
[INFO] optimize_disk_space: False
[INFO] orthophoto_compression: DEFLATE
[INFO] orthophoto_cutline: False
[INFO] orthophoto_kmz: False
[INFO] orthophoto_no_tiled: False
[INFO] orthophoto_png: False
[INFO] orthophoto_resolution: 5
[INFO] pc_classify: False
[INFO] pc_copc: False
[INFO] pc_csv: False
[INFO] pc_ept: True
[INFO] pc_filter: 5
[INFO] pc_las: False
[INFO] pc_quality: medium
[INFO] pc_rectify: False
[INFO] pc_sample: 0
[INFO] pc_skip_geometric: False
[INFO] primary_band: auto
[INFO] project_path: C:\WebODM\resources\app\apps\NodeODM\data\66102c90-90ea-458a-ac16-388a2606ae2a
[INFO] radiometric_calibration: none
[INFO] rerun: None
[INFO] rerun_all: False
[INFO] rerun_from: None
[INFO] rolling_shutter: False
[INFO] rolling_shutter_readout: 0
[INFO] sfm_algorithm: incremental
[INFO] sfm_no_partial: False
[INFO] skip_3dmodel: False
[INFO] skip_band_alignment: False
[INFO] skip_orthophoto: False
[INFO] skip_report: False
[INFO] sky_removal: False
[INFO] sm_cluster: None
[INFO] sm_no_align: False
[INFO] smrf_scalar: 1.25
[INFO] smrf_slope: 0.15
[INFO] smrf_threshold: 0.5
[INFO] smrf_window: 18.0
[INFO] split: 999999
[INFO] split_image_groups: None
[INFO] split_overlap: 150
[INFO] texturing_keep_unseen_faces: False
[INFO] texturing_single_material: False
[INFO] texturing_skip_global_seam_leveling: False
[INFO] tiles: False
[INFO] use_3dmesh: False
[INFO] use_exif: False
[INFO] use_fixed_camera_params: False
[INFO] use_hybrid_bundle_adjustment: False
[INFO] video_limit: 500
[INFO] video_resolution: 4000
[INFO] ==============
[INFO] Running dataset stage
[INFO] Loading dataset from: C:\WebODM\resources\app\apps\NodeODM\data\66102c90-90ea-458a-ac16-388a2606ae2a\images
[INFO] Loading 130 images
[INFO] Wrote images database: C:\WebODM\resources\app\apps\NodeODM\data\66102c90-90ea-458a-ac16-388a2606ae2a\images.json
[INFO] Found 130 usable images
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Normal dataset, will process all at once.
[INFO] Finished split stage
[INFO] Running merge stage
[INFO] Normal dataset, nothing to merge.
[INFO] Finished merge stage
[INFO] Running opensfm stage
[INFO] Maximum photo dimensions: 1280px
[INFO] Photo dimensions for feature extraction: 1280px
[INFO] CUDA drivers detected
[INFO] Altitude data detected, enabling it for GPS alignment
[INFO] [‘use_exif_size: no’, ‘flann_algorithm: KDTREE’, ‘feature_process_size: 1280’, ‘feature_min_frames: 10000’, ‘processes: 16’, ‘matching_gps_neighbors: 0’, ‘matching_gps_distance: 0’, ‘matching_graph_rounds: 50’, ‘optimize_camera_parameters: yes’, ‘reconstruction_algorithm: incremental’, ‘undistorted_image_format: tif’, ‘bundle_outlier_filtering_type: AUTO’, ‘sift_peak_threshold: 0.066’, ‘align_orientation_prior: vertical’, ‘triangulation_type: ROBUST’, ‘retriangulation_ratio: 2’, ‘matcher_type: FLANN’, ‘feature_type: DSPSIFT’, ‘use_altitude_tag: yes’, ‘align_method: auto’, ‘local_bundle_radius: 0’]
[INFO] Wrote reference_lla.json
[INFO] running “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm” detect_features “C:\WebODM\resources\app\apps\NodeODM\data\66102c90-90ea-458a-ac16-388a2606ae2a\opensfm”
d:\a\odm\odm\superbuild\build\pypopsift_deps\popsift-src\src\popsift\gauss_filter.cu:245
cudaMemcpyToSymbol failed for Gauss kernel initialization: unrecognized error code

[quinn-r88]

I'm also getting this same issue with ODM (non-web version). Looks very related to this: alicevision/popsift#160, and might imply that either the popsift dependency is out of date, or else not compiled for your graphics card. Do you know the CC code for your card? You can find it here: https://developer.nvidia.com/cuda-gpus

[cptcitrus]

I've just had this error on a new machine with a RTX 5080, CC 7.5. Error message attached. Desktop version of ODM.

Edit: I've now replicated it across three different imagery collections from two different UAVs. Will test on another machine next.

message.txt

[cturner8725]

I'm also getting this same issue with ODM (non-web version). Looks very related to this: alicevision/popsift#160, and might imply that either the popsift dependency is out of date, or else not compiled for your graphics card. Do you know the CC code for your card? You can find it here: https://developer.nvidia.com/cuda-gpus

I have a RTX 5080 like cptcitrus, so CC 7.5 as well. Looks like that code is also used for RTX 4000 and 3000 series as well. So, I would think this issue would be more prevalent if the CC code was the issue, right?

Edit: I've now replicated it across three different imagery collections from two different UAVs. Will test on another machine next.

I tested this across multiple data sets as well with different image counts and varied reduced image sizes. Same error. Have you tried restarting the task to see if the MVS stage uses your GPU?

[RomanIvn]

Same with WebODM web using DigitalOcean GPU droplet & 1XH100 running NodeODM in docker.

[INFO] DTM is turned on, automatically turning on point cloud classification
[INFO] Initializing ODM 3.5.5 - Sat Mar 15 22:03:28 2025
[INFO] ==============
[INFO] 3d_tiles: True
[INFO] align: None
[INFO] auto_boundary: False
[INFO] auto_boundary_distance: 0
[INFO] bg_removal: False
[INFO] boundary: {}
[INFO] build_overviews: False
[INFO] camera_lens: auto
[INFO] cameras: {}
[INFO] cog: True
[INFO] copy_to: None
[INFO] crop: 3
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: False
[INFO] dem_gapfill_steps: 3
[INFO] dem_resolution: 2.0
[INFO] dsm: True
[INFO] dtm: True
[INFO] end_with: odm_postprocess
[INFO] fast_orthophoto: False
[INFO] feature_quality: ultra
[INFO] feature_type: dspsift
[INFO] force_gps: False
[INFO] gcp: None
[INFO] geo: None
[INFO] gltf: True
[INFO] gps_accuracy: 3
[INFO] ignore_gsd: False
[INFO] matcher_neighbors: 0
[INFO] matcher_order: 0
[INFO] matcher_type: flann
[INFO] max_concurrency: 60
[INFO] merge: all
[INFO] mesh_octree_depth: 11
[INFO] mesh_size: 200000
[INFO] min_num_features: 10000
[INFO] name: 3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643
[INFO] no_gpu: False
[INFO] optimize_disk_space: False
[INFO] orthophoto_compression: DEFLATE
[INFO] orthophoto_cutline: False
[INFO] orthophoto_kmz: False
[INFO] orthophoto_no_tiled: False
[INFO] orthophoto_png: False
[INFO] orthophoto_resolution: 2.0
[INFO] pc_classify: True
[INFO] pc_copc: False
[INFO] pc_csv: False
[INFO] pc_ept: True
[INFO] pc_filter: 5
[INFO] pc_las: False
[INFO] pc_quality: ultra
[INFO] pc_rectify: False
[INFO] pc_sample: 0
[INFO] pc_skip_geometric: False
[INFO] primary_band: auto
[INFO] project_path: /var/www/data/3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643
[INFO] radiometric_calibration: none
[INFO] rerun: None
[INFO] rerun_all: False
[INFO] rerun_from: ['opensfm', 'openmvs', 'odm_filterpoints', 'odm_meshing', 'mvs_texturing', 'odm_georeferencing', 'odm_dem', 'odm_orthophoto', 'odm_report', 'odm_postprocess']
[INFO] rolling_shutter: False
[INFO] rolling_shutter_readout: 0
[INFO] sfm_algorithm: incremental
[INFO] sfm_no_partial: False
[INFO] skip_3dmodel: False
[INFO] skip_band_alignment: False
[INFO] skip_orthophoto: False
[INFO] skip_report: False
[INFO] sky_removal: False
[INFO] sm_cluster: None
[INFO] sm_no_align: False
[INFO] smrf_scalar: 1.25
[INFO] smrf_slope: 0.15
[INFO] smrf_threshold: 0.5
[INFO] smrf_window: 18.0
[INFO] split: 999999
[INFO] split_image_groups: None
[INFO] split_overlap: 150
[INFO] texturing_keep_unseen_faces: False
[INFO] texturing_single_material: False
[INFO] texturing_skip_global_seam_leveling: False
[INFO] tiles: False
[INFO] use_3dmesh: False
[INFO] use_exif: False
[INFO] use_fixed_camera_params: False
[INFO] use_hybrid_bundle_adjustment: False
[INFO] video_limit: 500
[INFO] video_resolution: 4000
[INFO] ==============
[INFO] Running dataset stage
[INFO] Loading dataset from: /var/www/data/3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643/images
[INFO] Loading images database: /var/www/data/3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643/images.json
[INFO] Found 45 usable images
[INFO] Coordinates file already exist: /var/www/data/3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643/odm_georeferencing/coords.txt
[INFO] Model geo file already exist: /var/www/data/3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643/odm_georeferencing/odm_georeferencing_model_geo.txt
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Normal dataset, will process all at once.
[INFO] Finished split stage
[INFO] Running merge stage
[INFO] Normal dataset, nothing to merge.
[INFO] Finished merge stage
[INFO] Running opensfm stage
[INFO] Maximum photo dimensions: 2048px
[INFO] Photo dimensions for feature extraction: 2048px
[INFO] nvidia-smi detected
[INFO] Altitude data detected, enabling it for GPS alignment
[INFO] ['use_exif_size: no', 'flann_algorithm: KDTREE', 'feature_process_size: 2048', 'feature_min_frames: 10000', 'processes: 60', 'matching_gps_neighbors: 0', 'matching_gps_distance: 0', 'matching_graph_rounds: 50', 'optimize_camera_parameters: yes', 'reconstruction_algorithm: incremental', 'undistorted_image_format: tif', 'bundle_outlier_filtering_type: AUTO', 'sift_peak_threshold: 0.066', 'align_orientation_prior: vertical', 'triangulation_type: ROBUST', 'retriangulation_ratio: 2', 'matcher_type: FLANN', 'feature_type: DSPSIFT', 'use_altitude_tag: yes', 'align_method: auto', 'local_bundle_radius: 0']
[INFO] Wrote reference_lla.json
[INFO] running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" detect_features "/var/www/data/3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643/opensfm"
/code/SuperBuild/build/pypopsift/_deps/popsift-src/src/popsift/gauss_filter.cu:245
cudaMemcpyToSymbol failed for Gauss kernel initialization: unrecognized error code

[cptcitrus]

The WebODM processing completed successfully when no-gpu flag was ticked. This is definitely a GPU issue. I can post the log if helpful but not today.

[quinn-r88]

It looks like there's maybe two different issues going on here.

ODM uses this version of PopSift, which hasnt been updated in a hot minute, and is hard-coded to only compile for some specific architectures here:

https://github.com/uav4geo/popsift/blob/c9719924d6c6926ca9e443b2766c07c4f99e8177/cmake/ChooseCudaCC.cmake#L68

So when RomanIvn and I try to run this on Digital Ocean's GPU stuff, the application just doesnt handle the newer hardware. Adding 90, to that line I linked should hypothetically fix it for us two, though I've not gotten that to work just yet.

But yeah for cpcitrus and cturner8725, idk what's going on, popsift is very explicitly built for your cards. Do you guys get anything like a build log? There should be a line in there that says Compiling for CUDA CCs: 35;50;52;60;61;70;75;80;86

[cptcitrus]

Okay, I've identified the issue. I was looking at the wrong CC on NVIDIA's website (RTX 5000 vs Geforce RTX 5080). My compute score is 10.0, which is above the level popsift can support. Can you confirm this theory?

From popsift code:

  # Create a list of possible CCs for each host processor.
  # This may require tuning: CUDA cards exist in AIX machines with POWER CPUs,
  # it is possible that non-Tegra ARM systems exist as well.
  # For now, this is my best guess.
  #
  set(TEGRA_SUPPORTED_PROCESSORS "armv71;arm;aarch64")
  set(OTHER_SUPPORTED_PROCESSORS "i686;x86_64;AMD64")

  set(CC_LIST_BY_SYSTEM_PROCESSOR "")
  if(CMAKE_SYSTEM_PROCESSOR IN_LIST OTHER_SUPPORTED_PROCESSORS)
    list(APPEND CC_LIST_BY_SYSTEM_PROCESSOR "20;21;30;35;50;52;60;61;70;75;80;86")
  endif()
  if(CMAKE_SYSTEM_PROCESSOR IN_LIST TEGRA_SUPPORTED_PROCESSORS)
    list(APPEND CC_LIST_BY_SYSTEM_PROCESSOR "32;53;62;72")
  endif()
  if(NOT CC_LIST_BY_SYSTEM_PROCESSOR)
    message(FATAL_ERROR "Unknown how to build for ${CMAKE_SYSTEM_PROCESSOR}")
  endif()

[quinn-r88]

yeah that looks right to me

Fix-wise: do you need need to use GPU acceleration? I might just skip it, at least for this specific stage. I don't think running SIFT on CPU is too expensive...

Also hey another Quinn in the wild!

[cptcitrus]

Another Quinn working in lidR even. We could start a club.

I think I will skip the GPU acceleration for now, which is too bad but not a big deal. Maybe popsift will be updated eventually.