👨‍🌾 test_generate_policy and test_generate_service flaky on macOS on foxy

[azeey]

Bug report

Required Info:

• Operating System:
  • macOs
• Installation type:
  • source
• Version or commit hash:
  • 0.9.4
• DDS implementation:
  • N/A
• Client library (if applicable):
  • rclpy

Steps to reproduce issue

test_generate_policy or test_generate_service tests fail consistently if ros2 daemon is running. On CI machines, the the tests are flaky, but fail most of the time (eg. history on Focal)

Expected behavior

All tests pass.

Actual behavior

test_generate_policy_topics fails with

test/sros2/commands/security/verbs/test_generate_policy.py:44: in test_generate_policy_topics
    assert cli.main(
E   AssertionError: assert 1 == 0
E    +  where 1 = <function main at 0x7f2914956a60>(argv=['security', 'generate_policy', '/tmp/tmpaxpo04wh/test-policy.xml'])
E    +    where <function main at 0x7f2914956a60> = cli.main
----------------------------- Captured stderr call -----------------------------
No nodes detected in the ROS graph. No policy file was generated.

and test_generate_policy_services fails with

test/sros2/commands/security/verbs/test_generate_policy.py:96: in test_generate_policy_services
    assert cli.main(
E   AssertionError: assert 1 == 0
E    +  where 1 = <function main at 0x7f2914956a60>(argv=['security', 'generate_policy', '/tmp/tmp2k4qonwa/test-policy.xml'])
E    +    where <function main at 0x7f2914956a60> = cli.main
----------------------------- Captured stderr call -----------------------------
No nodes detected in the ROS graph. No policy file was generated.

Additional information

Adding a small sleep right before calling cli.main in both tests fixes the tests for me locally. Alternatively, passing --no-daemon to generate_policy fixes the issue as well, but I'm not sure which one, if either, is a good option.

[mikaelarguedas]

Thanks for reporting, the test only fails in foxy and not in Nightly or Rolling jobs is that correct?

There was an oberhaul of the tests structure to remove the use of the daemon and make these tests less flaky #214, it was not backported to foxy because it was not considered a critical bug as this had no impact on the resulting debs.

[nuclearsandwich]

Thanks for reporting, the test only fails in foxy and not in Nightly or Rolling jobs is that correct?

It seems like this issue may have combined two separate sets of failures. The issue described is present in Foxy on Linux. The recent nightly and rolling builds don't seem to exhibit this failure.

it was not backported to foxy because it was not considered a critical bug as this had no impact on the resulting debs.

If it's a non-breaking change that can be backported doing so would increase the confidence when testing and producing Foxy patch releases.

---

Separately on macOS with rmw_connext_cpp there has been an sros2 test failure in two of the last three nightlies

• https://ci.ros2.org/view/nightly/job/nightly_osx_release/1981/testReport/junit/ros2param.ros2param.test/test_verb_list/test_verb_list/
• https://ci.ros2.org/view/nightly/job/nightly_osx_release/1983/testReport/junit/ros2param.ros2param.test/test_verb_list/test_verb_list/

I can open a separate issue for that if preferred.

[nuclearsandwich]

sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes has also failed on 3 out of 4 of the most recent Windows Debug nightlies.

• https://ci.ros2.org/view/nightly/job/nightly_win_deb/1911/testReport/sros2.test.sros2.commands.security.verbs/test_generate_policy_no_nodes/test_generate_policy_no_nodes/
• https://ci.ros2.org/view/nightly/job/nightly_win_deb/1912/testReport/sros2.test.sros2.commands.security.verbs/test_generate_policy_no_nodes/test_generate_policy_no_nodes/
• https://ci.ros2.org/view/nightly/job/nightly_win_deb/1914/testReport/sros2.test.sros2.commands.security.verbs/test_generate_policy_no_nodes/test_generate_policy_no_nodes/

[tfoote]

This continues to be an issue on Windows debug: https://ci.ros2.org/job/nightly_win_deb/1921/testReport/sros2.test.sros2.commands.security.verbs/test_generate_policy_no_nodes/test_generate_policy_no_nodes/history/

And it's also occurring on Linux foxy CI: https://build.ros2.org/job/Fci__nightly-fastrtps_ubuntu_focal_amd64/256/testReport/sros2.test.sros2.commands.security.verbs/test_generate_policy/test_generate_policy_topics/history/

[mikaelarguedas]

Foxy flakyness:
#254 is a backport of the testing overhaul to the foxy branch. Could you let us know if this addresses the issues noticed on Foxy?

---

I can open a separate issue for that if preferred.

Yes please let's open another issue for the nightly flakiness so that we dont mix the code versions to modify.

Separately on macOS with rmw_connext_cpp there has been an sros2 test failure in two of the last three nightlies

https://ci.ros2.org/view/nightly/job/nightly_osx_release/1981/testReport/junit/ros2param.ros2param.test/test_verb_list/test_verb_list/
https://ci.ros2.org/view/nightly/job/nightly_osx_release/1983/testReport/junit/ros2param.ros2param.test/test_verb_list/test_verb_list/

Wrong links ? these seem to be for ros2param

These tests have been causing issues for a long time as they rely on the ROS Graph to be clean. This seemed to be improved after #214 but from what I read here there seem to still be an issue on Windows Debug. Are there processes not terminating properly on these jobs ?

This continues to be an issue on Windows debug

These look like hanging issues. From #214 (comment) and looking at the job output, seems that most(/all?) ros2 cli related tests are disabled on windows. So maybe it's an issue from the tooling below the code in this repository? Should all ros2cli based tests just be disabled for Windows?

[mikaelarguedas]

@azeey did you have a chance to confirm if #254 fixes this issue ?

[ivanpauno]

There's a similar issue with the new rmw_connextdds, though the root cause might be different:

https://ci.ros2.org/view/nightly/job/nightly_linux_repeated/2249/testReport/sros2.test.sros2.commands.security.verbs/test_generate_policy/test_generate_policy/

[hidmic]

On that note, #260 may help. Launch related hangings may also have been solved in Galactic since ros2/launch#476. We haven't re-enabled tests on Windows though (still too flaky), so take it with a grain of salt.

[jacobperron]

I've merged #254. But also, macOS is no longer a supported platform for Foxy. So either way, I think we can close this ticket.