Pointcloud processor pipeline shows huge publishing latency with no errors

[siddharth-w]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.
• I'm convinced that this is not my fault but a bug.

Description

The crop_box_filter_self, crop_box_filter_mirror and distortion_corrector_node shows huge latency in subsequent message publish even though the
/pointcloud_raw_ex is getting published by the nebula_ros package with default frequency.
Following are the configuration and params with the behavior given in the "Actual behavior" section:

lidar.launch.xml :

vehicle_info.param.yaml :

vehicle_mirror.param.yaml

Expected behavior

Pointcloud pipeline has minimal latency

Actual behavior

As the crop_box_filter_self node doesnt take any other input so it shouldnt be related to the anything other than pointcloud input. Below is the actual behaviour:

pipeline_latency_ms :

ros2 topic echo /sensing/lidar/top/crop_box_filter_self/debug/pipeline_latency_ms
                                                                
stamp:
  sec: 1737126653
  nanosec: 337972075
data: 518.684237
---
stamp:
  sec: 1737126655
  nanosec: 381116577
data: 562.35845
---
stamp:
  sec: 1737126658
  nanosec: 205620609
data: 587.268885
---

processing_time_ms :

ros2 topic echo /sensing/lidar/top/crop_box_filter_self/debug/processing_time_ms 

stamp:
  sec: 1737126653
  nanosec: 337906384
data: 5.598
---
stamp:
  sec: 1737126655
  nanosec: 381072219
data: 5.79
---
stamp:
  sec: 1737126658
  nanosec: 205581197
data: 6.181
---

published timestamp

ros2 topic echo /sensing/lidar/top/self_cropped/pointcloud_ex/debug/published_time 
header:
  stamp:
    sec: 1737126652
    nanosec: 819285760
  frame_id: ''
published_stamp:
  sec: 1737126653
  nanosec: 338033688
---
header:
  stamp:
    sec: 1737126654
    nanosec: 818756096
  frame_id: ''
published_stamp:
  sec: 1737126655
  nanosec: 381204125
---
header:
  stamp:
    sec: 1737126657
    nanosec: 618349824
  frame_id: ''
published_stamp:
  sec: 1737126658
  nanosec: 205685058
---

The same behavior is carried forward till localization/util/downsample/pointcloud

Please help in an elaborated way :(

Steps to reproduce

1. Connect and configure the sensors and vehicle params
2. Launch autoware.launch.xml

Versions

OS: Ubuntu 22.04.5 LTS x86_64
ROS 2: Humble
Autoware: 0.39.0
Lidar: Velodyne VLP16
GPS: Ublox F9P
IMU: Zed2 camera IMU
CPU: Intel Xeon W-1390P (16) @ 5.100GHz
GPU: NVIDIA RTX A5000
GPU: Intel RocketLake-S GT1 [UHD Graphics P750]
Memory: 7548MiB / 64026MiB

Possible causes

No response

Additional context

No response

[xmfcx]

Hi, Could you make sure DDS is configured correctly? Please follow to check:

• https://autowarefoundation.github.io/autoware-documentation/main/installation/additional-settings-for-developers/network-configuration/dds-settings/

Also can you check if the nebula driver and the crobbox filter are in the same composable node container?

For this, you can run ros2 component list to see the containers and composable nodes.

You may also share the ros2 bag, this will make it easier to debug.

Also can you set the FOV to full 360 degrees, without limiting to see if it might have caused the issue?

[siddharth-w]

Hi, Thanks for looking into it.

This is the component list, and nebula driver and the cropbox filter are indeed in the same container.

ros2 component list
/system/component_state_monitor/container
  1  /system/component_state_monitor/component
/map/map_container
  1  /map/pointcloud_map_loader
  2  /map/lanelet2_map_loader
  3  /map/lanelet2_map_visualization
  4  /map/vector_map_tf_generator
/system/mrm_comfortable_stop_operator/mrm_comfortable_stop_operator_container
  1  /system/mrm_comfortable_stop_operator
/system/mrm_emergency_stop_operator/mrm_emergency_stop_operator_container
  1  /system/mrm_emergency_stop_operator
/control/control_container
  1  /control/glog_control_container_component
  2  /control/external_cmd_converter
  3  /control/external_cmd_selector
  4  /control/trajectory_follower/controller_node_exe
  5  /control/autoware_shift_decider
  6  /control/vehicle_cmd_gate
  7  /control/autoware_operation_mode_transition_manager
/zed/zed_container
  1  /zed/zed_node
/sensing/lidar/top/pointcloud_preprocessor/pointcloud_container
  1  /sensing/lidar/top/glog_component
  2  /sensing/lidar/top/velodyne_ros_wrapper_node
  3  /sensing/lidar/top/crop_box_filter_self
  4  /sensing/lidar/top/crop_box_filter_mirror
  5  /sensing/lidar/top/distortion_corrector_node
  6  /sensing/lidar/top/ring_outlier_filter
/adapi/container
  1  /adapi/node/autoware_state
  2  /adapi/node/diagnostics
  3  /adapi/node/fail_safe
  4  /adapi/node/heartbeat
  5  /adapi/node/interface
  6  /adapi/node/localization
  7  /adapi/node/motion
  8  /adapi/node/operation_mode
  9  /adapi/node/perception
  10  /adapi/node/planning
  11  /adapi/node/routing
  12  /adapi/node/vehicle
  13  /adapi/node/vehicle_info
  14  /adapi/node/vehicle_door
/control/control_check_container
  1  /control/glog_control_check_container_component
  2  /control/trajectory_follower/lane_departure_checker_node
  3  /control/control_validator
  5  /control/collision_detector
/system/system_monitor/system_monitor/system_monitor_container
  1  /system/system_monitor/cpu_monitor
  2  /system/system_monitor/mem_monitor
  3  /system/system_monitor/net_monitor
  4  /system/system_monitor/ntp_monitor
  5  /system/system_monitor/process_monitor
  6  /system/system_monitor/hdd_monitor
  7  /system/system_monitor/gpu_monitor
  8  /system/system_monitor/voltage_monitor
/pointcloud_container
  1  /localization/util/crop_box_filter_measurement_range
  2  /sensing/lidar/concatenate_data
  3  /localization/util/voxel_grid_downsample_filter
  4  /localization/util/random_downsample_filter
/autoware_api/external/rtc_controller/container
  1  /autoware_api/external/rtc_controller/node

I might be hallucinating, but I think changing the DDS settings made it a bit faster, although its still painfully slow. I did change the FOV, cloud_min_angle=0 to cloud_max_angle=359 and the result was still the same.
I am attaching the ros2 bag as a zip file: ros2 bag