Robot Stuck in "Waiting for Traffic" State Despite No Blockage

[alex-roba]

I've observed an issue where a robot unnecessarily waits for traffic at a waypoint, even though its destination is clear and unblocked. The robot remains stuck in the "Waiting for Traffic" state indefinitely, until another robot moves to a different location. At that point, the waiting robot resumes its operation.

I've attached a video demonstrating the behavior. In the video, pay close attention to Robot 1, which exhibits the issue. It stops and waits unnecessarily without any apparent reason for the delay.

unncceary_wait.-.Trim.online-video-cutter.com.1.mp4

[mxgrey]

Thanks for catching and reporting this. It appears to be a bug in either the traffic dependency system or the traffic schedule update system. If you're able to provide a package (i.e. the map file and some launch files) that reliably reproduce the bug, I'll dig in to identify the cause and figure out a fix.

[alex-roba]

Hi @mxgrey,

I was able to reliably reproduce the bug in the office world. Below is the step-by-step procedure in Jazzy:

1. Launch the office world using:
   ros2 launch rmf_demos_gz office.launch.xml

2. Send a patrol task for robot 1 to the lounge using:
   ros2 run rmf_demos_tasks dispatch_patrol -F tinyRobot -R tinyRobot1 -p lounge -n 1 --use_sim_time

3. Once robot 1 passes the patrol_D1 waypoint, send a patrol task for robot 2 to the lounge waypoint using:
   ros2 run rmf_demos_tasks dispatch_patrol -F tinyRobot -R tinyRobot2 -p lounge -n 1 --use_sim_time

You will notice that tinyRobot1 is unnecessarily waiting at the waypoint before the lounge, even though tinyRobot2 has already completed its task, arrived at the lounge, and is parked. This wait is eventually interrupted by the excessively delayed behavior, which lasts around 30 seconds.

I am not using the reservation node here, but the same bug is also observed when the reservation node is enabled.

Let me know if you need more details!

wait.delay.-.Trim.online-video-cutter.com.mp4

[mxgrey]

Thanks for figuring out the minimum reproducible example, I wouldn't have been able to figure out the cause of the bug without this. I've opened a PR to fix it here: open-rmf/rmf_traffic#113

If you get a chance to try out that PR, let me know how it goes, especially if you run into any other cases of traffic waiting incorrectly.

[mxgrey]

I am not using the reservation node here, but the same bug is also observed when the reservation node is enabled.

On a separate note, if you see the exact same behavior with and without the reservation system running, then I think you aren't enabling reservations for your fleet. That behavior doesn't get turned on by default since it would be a major behavioral change for prior users.

[alex-roba]

Hi @mxgrey,

I was able to reliably reproduce the bug in the office world. Below is the step-by-step procedure in Jazzy:

Launch the office world using:
ros2 launch rmf_demos_gz office.launch.xml

Send a patrol task for robot 1 to the lounge using:
ros2 run rmf_demos_tasks dispatch_patrol -F tinyRobot -R tinyRobot1 -p lounge -n 1 --use_sim_time

Once robot 1 passes the patrol_D1 waypoint, send a patrol task for robot 2 to the lounge waypoint using:
ros2 run rmf_demos_tasks dispatch_patrol -F tinyRobot -R tinyRobot2 -p lounge -n 1 --use_sim_time

You will notice that tinyRobot1 is unnecessarily waiting at the waypoint before the lounge, even though tinyRobot2 has already completed its task, arrived at the lounge, and is parked. This wait is eventually interrupted by the excessively delayed behavior, which lasts around 30 seconds.

I am not using the reservation node here, but the same bug is also observed when the reservation node is enabled.

Let me know if you need more details!

[osama-z-salah]

@mxgrey I wanted to follow up and ask if you’re planning to work on fixing this issue soon. I’d love to help debug if possible—do you have any insights or suggestions on where I should start looking?
Looking forward to your thoughts!

[mxgrey]

It's very near the top of my priority list. I expect I'll be able to make time for it this week. But if anyone wants to investigate sooner, I do know where I'd look to track down the problem.

• [1] is where we check if the traffic is ready for us to pass. If this isn't getting triggered when it should, or if it's receiving the wrong index numbers for the other robots, then the waiting robot will not know to go.
• [2] and [3] are where a robot updates what progress it has made along its path. If these lines aren't getting triggered when they should or if they're sending out the wrong index numbers, then other robots waiting on them won't know to proceed.

I would put gratuitous debug statements in both of these locations and then run a scenario where the issue occurs. Then try to narrow down where exactly the problem is happening based on which part isn't working the way it's supposed to.

[mxgrey]

The problem should be fixed by this PR: open-rmf/rmf_traffic#113

[alex-roba]

@mxgrey Thanks for your fix! I tried your PR (open-rmf/rmf_traffic#113), and it has reduced some of the delay. However, there are still occasional delays that appear unpredictably.

In the video below, I send both robots to the lounge sequentially with using_parking_reservation set to true. This means one robot should wait at the waypoint "wait", which I placed near the lounge.

As you can see, for the first attempt, the robots proceed without unnecessary waiting. However, on the second attempt, at 00:57 in the video, tinyRobot1 experiences an unexpected delay near to the negotiation area. I'm not sure why this happens, as the task remains the same across all two attempts.

If you'd like, I can share the configuration and navigation graph of my experiment.

Untitled.video.-.Made.with.Clipchamp.mp4

[alex-roba]

@mxgrey this is the i am sending the tasks sequentially ros2 run rmf_demos_tasks dispatch_go_to_place -p lounge -F tinyRobot -R tinyRobot2 --use_sim_time && ros2 run rmf_demos_tasks dispatch_go_to_place -p lounge -F tinyRobot -R tinyRobot1 --use_sim_time

[alex-roba]

@mxgrey Were you able to reproduce that scenario? If not, I can investigate further to find more reliable cases where the unnecessary wait occurs.

[mxgrey]

Yep, I'm managing to reproduce it and in the process of debugging it right now!

It's taking longer to debug than it normally would because the issue almost vanishes as I add in debug output, but I'm gradually narrowing the possible causes down.

[mxgrey]

I've tracked down the cause and opened a PR to fix it: open-rmf/rmf_traffic#115

Thanks again for coming up with the reproducible example. Once again I wouldn't have been able to track down the cause without it.

[mxgrey]

The fix is merged now. Let us know if you run into any new situations where the dependency lingers longer than it should.