Running a failing test "succeeds" with exit code 0 and "Invalid XML" message

[mortenfyhn]

I'm not sure if this repo will see activity now after ROS 1 end-of-life, but in any case:

If I have any failing catch_ros test, running it gives an error "Invalid XML in result file" and then exits with code zero (!). This of course means that it's easy to miss a broken test, because CI jobs etc stay green.

Steps to reproduce:

• ROS Noetic on Ubuntu 20.04.6
• Run latest catch_ros release 0.5.0 (or latest master e371074 built from source)
• Write a catch_ros test that fails, e.g. based on the example in the readme here, but setting (1 + 1) == 3
• Run it with catkin_make run_tests or catkin test
• You get the normal "failed test" Catch2 output, but the exit code is zero, and you get the "Invalid XML in result file" as part of the output.

I did a quick git bisect and found that the problem started in 724ab21 (between 0.4.0 and 0.5.0). 724ab21 updates Catch2 from v2.4.2 to v2.11.3. Repeating the above on 0.4.0 works as expected.

Example output, note "Invalid XML" near bottom:

$ catkin_make run_tests
Base path: /home/morten/workspace/debug_catch_ws
Source space: /home/morten/workspace/debug_catch_ws/src
Build space: /home/morten/workspace/debug_catch_ws/build
Devel space: /home/morten/workspace/debug_catch_ws/devel
Install space: /home/morten/workspace/debug_catch_ws/install
####
#### Running command: "make cmake_check_build_system" in "/home/morten/workspace/debug_catch_ws/build"
####
####
#### Running command: "make run_tests -j16 -l16" in "/home/morten/workspace/debug_catch_ws/build"
####
[ 50%] Built target catch_ros_standalone
Removing test result files from '/home/morten/workspace/debug_catch_ws/build/test_results/debug_catch'
- removing '/home/morten/workspace/debug_catch_ws/build/test_results/debug_catch/catch-my_standalone_test.xml'
Scanning dependencies of target my_standalone_test
[ 50%] Built target clean_test_results_debug_catch
[ 66%] Building CXX object debug_catch/CMakeFiles/my_standalone_test.dir/test/my_test.cpp.o
[ 83%] Linking CXX executable /home/morten/workspace/debug_catch_ws/devel/lib/debug_catch/my_standalone_test
[100%] Built target my_standalone_test
[100%] Built target tests
-- run_tests.py: execute commands
  /home/morten/workspace/debug_catch_ws/devel/lib/debug_catch/my_standalone_test -r ros_junit -o /home/morten/workspace/debug_catch_ws/build/test_results/debug_catch/catch-my_standalone_test.xml

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
my_standalone_test is a Catch v2.13.7 host application.
Run with -? for options

-------------------------------------------------------------------------------
test_case
-------------------------------------------------------------------------------
/home/morten/workspace/debug_catch_ws/src/debug_catch/test/my_test.cpp:3
...............................................................................

/home/morten/workspace/debug_catch_ws/src/debug_catch/test/my_test.cpp:5: FAILED:
  REQUIRE( (1 + 1) == 3 )
with expansion:
  2 == 3

===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed

-- run_tests.py: verify result "/home/morten/workspace/debug_catch_ws/build/test_results/debug_catch/catch-my_standalone_test.xml"
Invalid XML in result file '/home/morten/workspace/debug_catch_ws/build/test_results/debug_catch/catch-my_standalone_test.xml' (even after trying to tidy it): not well-formed (invalid token): line 2, column 0
[100%] Built target _run_tests_debug_catch_catch_my_standalone_test
[100%] Built target _run_tests_debug_catch_catch
[100%] Built target _run_tests_debug_catch
[100%] Built target run_tests

Here's what the invalid XML looks like:

<?xml version="1.0" encoding="UTF-8"?>
?[0;37m?[0m?[0m?[0m?[0;37m?[0m?[0;37m?[0m?[1;31m?[0m?[0;36m?[0m?[1;33m?[0m<testsuites errors="0" failures="1" tests="1">
  <testsuite name="my_standalone_test" errors="0" failures="1" tests="1" hostname="tbd" package="debug_catch" time="0" timestamp="tbd">
    <testcase classname="debug_catch.my_standalone_test" name="test_case" time="0.000000">
      <failure message="2 == 3" type="REQUIRE">
at /home/morten/workspace/debug_catch_ws/src/debug_catch/test/my_test.cpp:5
      </failure>
    </testcase>
    <system-out/>
    <system-err/>
  </testsuite>
</testsuites>?[1;31m?[0m?[1;33m?[0m?[0;32m?[0m?[0;37m?[0m?[1;31m?[0m?[0;37m?[0m?[1;31m?[0m

And here's the XML from the same setup but running catch_ros 0.4.0:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites errors="0" failures="1" tests="1">
  <testsuite name="my_standalone_test" errors="0" failures="1" tests="1" hostname="tbd" package="debug_catch" time="0" timestamp="tbd">
    <testcase classname="debug_catch.my_standalone_test" name="test_case" time="0.000000">
      <failure message="2 == 3" type="REQUIRE">
at /home/morten/workspace/debug_catch_ws/src/debug_catch/test/my_test.cpp:5
      </failure>
    </testcase>
    <system-out/>
    <system-err/>
  </testsuite>
</testsuites>