Sporadic test fail from kafkaClusterKraftModeWithMultipleControllers

[k-wall]

Please use this to only for bug reports. For questions or when you need help, you can use the GitHub Discussions or use the community Slack chat.

Describe the bug

Whilst trying to reproduce another test failure (#182) , I noticed the following test is sporadically failing, about 1 time in 10 for me.

2023-09-21 11:01:46 WARN  awaitility-thread io.kroxylicious.testing.kafka.invm.InVMKafkaCluster:207 - failed to start server due to: Received a fatal error while waiting for the controller to acknowledge that we are caught up
2023-09-21 11:01:48 WARN  awaitility-thread io.kroxylicious.testing.kafka.invm.InVMKafkaCluster:208 - anon: //0.0.0.0:12718, client: //0.0.0.0:28508, controller: //0.0.0.0:12194, interBroker: //0.0.0.0:28508,
[ERROR] Tests run: 33, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 199.9 s <<< FAILURE! -- in io.kroxylicious.testing.kafka.KafkaClusterTest
[ERROR] io.kroxylicious.testing.kafka.KafkaClusterTest.kafkaClusterKraftModeWithMultipleControllers -- Time elapsed: 31.48 s <<< ERROR!
org.awaitility.core.ConditionTimeoutException: Condition with io.kroxylicious.testing.kafka.invm.InVMKafkaCluster was not fulfilled within 30 seconds.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
	at io.kroxylicious.testing.kafka.invm.InVMKafkaCluster.tryToStartServerWithRetry(InVMKafkaCluster.java:198)
	at io.kroxylicious.testing.kafka.invm.InVMKafkaCluster.lambda$start$5(InVMKafkaCluster.java:187)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
	at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
Caused by: java.util.concurrent.TimeoutException
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
	at org.awaitility.core.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:101)
	at org.awaitility.core.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:81)
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:103)
	... 16 more

To Reproduce
Steps to reproduce the behavior:

1. Run mvn verify in a loop

while mvn verify
do
i=$((i+1)); echo done $i
done

Expected behavior
No sporadic failure.

Logs

Attach or copy and paste relevant logs.

Additional context
Add any other context about the problem here.