[CX_HARDENING] - Audit and Improve test_with_failures_one

[jparr721]

What is this task and why do we need to work on it?

The test_with_failures_one integration test is currently always deterministically failing the exact same node. There exists a linked issue in the test that was originally preventing us from shutting down any node. It appears that this has been resolved, and we should take steps to remove the determinism from this test to enhance the reliability of the results.

What work will need to be done to complete this task?

• Verify that, indeed, Update message broadcasting to avoid hanging #1567 is closed and resolved.
• Add a new test that removes the hard-coded node number and instead randomly selects from a range of valid nodes (if Update message broadcasting to avoid hanging #1567 is not resolved, adjust this accordingly).
• Look into adding non determinism to the Spinning Task description as well.

Are there any other details to include?

If desired, we could have a test for each node (though this might take a long time to complete).

What are the acceptance criteria to close this issue?

The existing test still passes, and the new test also passes.

Branch work will be merged to (if not the default branch)

No response