autoware_multi_object_tracker has a significant delay when publishing the results.

[vollkornholztuer]

My issue

Hey all, i encountered the following issue:

For context: I'm using autoware_lidar_centerpoint in combination with autoware_multi_object_tracker to detect and track vehicles in a sequence of pointclouds from a bag-file (.db3).

I encountered the issue, that the published/displayed results from autoware_multi_object_tracker are very delayed as seen in the video below:
Screencast from 01-30-2025 02:46:56 PM.webm
(The playback of the rosbag file starts at 5 seconds video playback time)

this is how the corresponding rqt-graph looks like as of today, when i noticed the issue:

The interesting thing is, i also managed to run said chain, where the results were instantly published by autoware_multi_object_tracker, as seen in the video below:
https://github.com/user-attachments/assets/4dc34a3f-0aa6-496a-a778-8236ade690be

I sadly do not have a picture of the rqt-node graph, but it should have been the same. Anything else woudln't make sense in my opinion.

Reproduction of the issue

This is how you can reproduce my steps:

1. In an empty terminal, launch autoware_multi_object_tracker with the command: ros2 launch autoware_multi_object_tracker multi_object_tracker.launch.xml
2. In another emtpy terminal, launch autoware_lidar_centerpoint with a remap for autoware_multi_object_tracker. This is the command: ros2 launch autoware_lidar_centerpoint lidar_centerpoint.launch.xml output/objects:=/perception/object_recognition/detection/objects
3. In another empty terminal, playback the .db3 file, with a remap for autoware_lidar_centerpoint with the following command: `ros2 bag play "/path/to/bagfile.db3" --remap /pointcloud2:=/sensing/lidar/pointcloud

Related files that might be important

About the launch.xml and the param.yaml files, these are the contents:

lidar_centerpoint.launch.xml

<?xml version="1.0"?>
<launch>
  <arg name="input/pointcloud" default="/sensing/lidar/pointcloud"/>
  <arg name="output/objects" default="detected_objects"/>
  <arg name="data_path" default="$(env HOME)/autoware_data" description="packages data and artifacts directory path"/>
  <!--
  <arg name="model_name" default="centerpoint_tiny" description="options: `centerpoint` or `centerpoint_tiny`"/>
  -->
  <arg name="model_name" default="centerpoint" description="options: `centerpoint` or `centerpoint_tiny`"/>
  <arg name="model_path" default="$(var data_path)/lidar_centerpoint"/>
  <arg name="model_param_path" default="$(find-pkg-share autoware_lidar_centerpoint)/config/$(var model_name).param.yaml"/>
  <arg name="ml_package_param_path" default="$(var model_path)/$(var model_name)_ml_package.param.yaml"/>
  <arg name="class_remapper_param_path" default="$(var model_path)/detection_class_remapper.param.yaml"/>
  <arg name="build_only" default="false" description="shutdown node after TensorRT engine file is built"/>
  <arg name="use_pointcloud_container" default="false" description="use pointcloud_container"/>
  <arg name="pointcloud_container_name" default="pointcloud_container" description="pointcloud_container name"/>

  <group unless="$(var use_pointcloud_container)">
    <node pkg="autoware_lidar_centerpoint" exec="autoware_lidar_centerpoint_node" name="lidar_centerpoint" output="screen">
      <remap from="~/input/pointcloud" to="$(var input/pointcloud)"/>
      <remap from="~/output/objects" to="$(var output/objects)"/>
      <param from="$(var model_param_path)" allow_substs="true"/>
      <param from="$(var ml_package_param_path)" allow_substs="true"/>
      <param from="$(var class_remapper_param_path)"/>
      <param name="build_only" value="$(var build_only)"/>
      <!--
      <param name="use_sim_time" value="true"/>
      -->
    </node>
  </group>
  <group if="$(var use_pointcloud_container)">
    <load_composable_node target="$(var pointcloud_container_name)">
      <composable_node pkg="autoware_lidar_centerpoint" plugin="autoware::lidar_centerpoint::LidarCenterPointNode" name="lidar_centerpoint">
        <remap from="~/input/pointcloud" to="$(var input/pointcloud)"/>
        <remap from="~/output/objects" to="$(var output/objects)"/>
        <param from="$(var model_param_path)" allow_substs="true"/>
        <param from="$(var ml_package_param_path)" allow_substs="true"/>
        <param from="$(var class_remapper_param_path)"/>
        <param name="build_only" value="$(var build_only)"/>
        <!--
        <param name="use_sim_time" value="true"/>
        -->
      </composable_node>
    </load_composable_node>
  </group>
</launch>

centerpoint.param.yaml (autoware_lidar_centerpoint)

/**:
  ros__parameters:
    # weight files
    encoder_onnx_path: "$(var model_path)/pts_voxel_encoder_$(var model_name).onnx"
    encoder_engine_path: "$(var model_path)/pts_voxel_encoder_$(var model_name).engine"
    head_onnx_path: "$(var model_path)/pts_backbone_neck_head_$(var model_name).onnx"
    head_engine_path: "$(var model_path)/pts_backbone_neck_head_$(var model_name).engine"
    trt_precision: fp16
    cloud_capacity: 2000000
    post_process_params:
      # post-process params
      circle_nms_dist_threshold: 0.5
      iou_nms_target_class_names: ["CAR"]
      iou_nms_search_distance_2d: 10.0
      iou_nms_threshold: 0.1
      score_threshold: 0.35
      yaw_norm_thresholds: [0.3, 0.3, 0.3, 0.3, 0.0]
    densification_params:
      world_frame_id: base_link
      num_past_frames: 1

multi_object_tracker_node.param.yaml (autoware_multi_object_tracker)

/**:
  ros__parameters:
    # default tracker models for each class
    car_tracker: "multi_vehicle_tracker"
    truck_tracker: "multi_vehicle_tracker"
    bus_tracker: "multi_vehicle_tracker"
    trailer_tracker: "multi_vehicle_tracker"
    pedestrian_tracker: "pedestrian_and_bicycle_tracker"
    bicycle_tracker: "pedestrian_and_bicycle_tracker"
    motorcycle_tracker: "pedestrian_and_bicycle_tracker"

    # default tracker node parameters
    publish_rate: 10.0
    world_frame_id: base_link # original was "map"
    enable_delay_compensation: false

    # debug parameters
    publish_processing_time: false
    publish_tentative_objects: false
    publish_debug_markers: false
    diagnostics_warn_delay: 0.5      # [sec]
    diagnostics_error_delay: 1.0     # [sec]

multi_object_tracker.launch.xml (autoware_multi_object_tracker)

PS: It doesn't change anything if i use `detected_objects` or `lidar_centerpoint` as the selected input channel. The issue persists, i just have to remap accordingly.

<?xml version="1.0" encoding="UTF-8"?>
<launch>
  <!--
  <arg name="selected_input_channels" default="['detected_objects']"/>
  -->
  <arg name="selected_input_channels" default="['lidar_centerpoint']"/>
  
  <arg name="output" default="objects"/>
  <arg name="tracker_setting_path" default="$(find-pkg-share autoware_multi_object_tracker)/config/multi_object_tracker_node.param.yaml"/>
  <arg name="data_association_matrix_path" default="$(find-pkg-share autoware_multi_object_tracker)/config/data_association_matrix.param.yaml"/>
  <arg name="input_channels_path" default="$(find-pkg-share autoware_multi_object_tracker)/config/input_channels.param.yaml"/>

  <node pkg="autoware_multi_object_tracker" exec="multi_object_tracker_node" name="multi_object_tracker" output="both">
    <param name="selected_input_channels" value="$(var selected_input_channels)"/>
    <remap from="output" to="$(var output)"/>
    <param from="$(var tracker_setting_path)"/>
    <param from="$(var data_association_matrix_path)"/>
    <param from="$(var input_channels_path)"/>
    <!--
    <param name="use_sim_time" value="true"/>
    -->
  </node>
</launch>

If you can help me figure this out, i would be very thankful. Thank you very much for reading.

[amadeuszsz]

@vollkornholztuer
There are multiple reason of high latency:

• Not configured DDS (refer to our docs). Make sure your environment variables are valid in all used terminal windows (or just use one launch file).
• Inter-process communication. For topics with large data better use inter-process communication (composable nodes in one ROS container).

I'm not sure if I understood your problem correctly, but from first video look like there was long startup time only. You need to check console logs to see if any initialization process is ongoing before publishing first msg.

[vollkornholztuer]

@amadeuszsz

I will look into the possible solutions. Thank you very much.

[vollkornholztuer]

By trying to find out more about my problem, i decided to echo the topic /diagnostics published by autoware_multi_object_tracker and this is the output of it:

---
header:
  stamp:
    sec: 1738331792
    nanosec: 611213257
  frame_id: ''
status:
- level: "\x02"
  name: 'multi_object_tracker: Perception delay check from original header stamp'
  message: Detection delay is over error threshold.
  hardware_id: multi_object_tracker
  values:
  - key: Detection delay
    value: '1.57991e+07'
---

What does this specifically mean? I also get this message with use_sim_time set to True. How would i go about fixing this?

EDIT:

About your previous answer and the DDS configuration, this is what i have configured:

# Increase the maximum receive buffer size for network packets
net.core.rmem_max=2147483647  # 2 GiB, default is 208 KiB

# IP fragmentation settings
net.ipv4.ipfrag_time=3  # in seconds, default is 30 s
net.ipv4.ipfrag_high_thresh=134217728  # 128 MiB, default is 256 KiB

this is the contents of my cyclonedds.xml file:

<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
  <Domain Id="any">
    <General>
      <Interfaces>
        <NetworkInterface autodetermine="false" name="lo" priority="default" multicast="default" />
      </Interfaces>
      <AllowMulticast>default</AllowMulticast>
      <MaxMessageSize>65500B</MaxMessageSize>
    </General>
    <Internal>
      <SocketReceiveBufferSize min="10MB"/>
      <Watermarks>
        <WhcHigh>500kB</WhcHigh>
      </Watermarks>
    </Internal>
  </Domain>
</CycloneDDS>

Mind that this configuration was set while installing autoware.

[amadeuszsz]

In log you can see huge delay because of difference between system time and rosbag time. Make sure all your nodes have defined use_sim_time as true and add --clock flag to rosbag play command. After that you can validate if all nodes publish messages to topics with same timestamps.