Add tests to make sure our test framework tolerates maven reruns even with test profiles

[holly-cummins]

Reproducer for #46048

As expected, this passes on main. To my surprise, it also passes on the latest version of #34681. I was assuming I had the test wrong or wasn't testing the correct scenario (after all, never trust a test you haven't seen fail). So I checked against last week's version of #34681, and these tests failed.

It appears a miracle has happened, and #46048 has been fixed sometime in the last week, without needing any maven changes. When I looked at this problem last time, the tests were not reloaded by the FacadeClassLoader on the second test run, so they'd run in the old classloader against a closed Quarkus application, and that worked about as well as you'd expect. This time, I've confirmed that on the second run the tests do get freshly loaded, and so everything works.

I can also see in CI that failing LGTM tests are being rerun, and they're failing in the LGTM code each time, not in the test framework code. So that's a useful extra confirmation, si

I'm not 100% sure what fixed it, but I think it's perhaps a combination of @aloubyansky's suggestions about cleaner test file discovery, and my work to close classloaders and leak less.

[holly-cummins]

Rebasing in the hopes that the problem with the build reports has been fixed upstream.

[gsmet]

There's something odd here, the build report is failing with the flake message being null, which is the very very first time I actually see that.

[holly-cummins]

There's something odd here, the build report is failing with the flake message being null, which is the very very first time I actually see that.

Ohh, that's got to be related to my changes, then. I saw Qute and assumed it wasn't me. Although I don't know why it would be happening, because the failure is just an ordinary test failure. I'll have a look to see if there's some exception message I've failed to fill in somewhere.

I guess it also means my test will turn up on every flake report, which is a bit irritating, even if I make sure the message is "deliberate failure" or something informative. I wonder if there's a property I can set to exclude it?

[holly-cummins]

I think the problem might have been caused by failing with thrown InvocationTargetExceptions, which (maybe?) have a null message. I've adjusted the fake-flaky test to unwrap the exception, and squashed.

[gsmet]

I deployed a fix to both the Bot and the Build Reporter Action.

[quarkus-bot]

Status for workflow Quarkus CI

This is the status report for running Quarkus CI on commit 39c51b8.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

You can consult the Develocity build scans.

---

Flaky tests - Develocity

⚙️ Maven Tests - JDK 17

📦 integration-tests/maven/target/test-classes/projects/test-flaky-test-multiple-profiles-processed

✖ org.acme.FlakyHelloResourceWithAnotherProfileTest.testHelloEndpoint - History

• deliberate failure run 1 - org.acme.FlakingException

org.acme.FlakingException: deliberate failure run 1
	at org.acme.FlakyHelloResourceWithAnotherProfileTest.testHelloEndpoint(FlakyHelloResourceWithAnotherProfileTest.java:35)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:960)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:810)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

📦 integration-tests/maven/target/test-classes/projects/test-flaky-test-single-profile-processed

✖ org.acme.FlakyHelloResourceTest.testHelloEndpoint - History

• deliberate failure run 1 - org.acme.FlakingException

org.acme.FlakingException: deliberate failure run 1
	at org.acme.FlakyHelloResourceTest.testHelloEndpoint(FlakyHelloResourceTest.java:31)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:960)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:810)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

---

⚙️ Maven Tests - JDK 17 Windows

📦 integration-tests/maven/target/test-classes/projects/test-flaky-test-multiple-profiles-processed

✖ org.acme.FlakyHelloResourceWithAnotherProfileTest.testHelloEndpoint - History

• deliberate failure run 1 - org.acme.FlakingException

org.acme.FlakingException: deliberate failure run 1
	at org.acme.FlakyHelloResourceWithAnotherProfileTest.testHelloEndpoint(FlakyHelloResourceWithAnotherProfileTest.java:35)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:960)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:810)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

📦 integration-tests/maven/target/test-classes/projects/test-flaky-test-single-profile-processed

✖ org.acme.FlakyHelloResourceTest.testHelloEndpoint - History

• deliberate failure run 1 - org.acme.FlakingException

org.acme.FlakingException: deliberate failure run 1
	at org.acme.FlakyHelloResourceTest.testHelloEndpoint(FlakyHelloResourceTest.java:31)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:960)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:810)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[holly-cummins]

Either your fix or my unwrapping of the exception sorted it out, @gsmet. Do you think we should try and exclude those failures from the build report? Otherwise that's going to be annoyingly noisy.

[gsmet]

Ah, my, yes, this is going to be very annoying.

Let's not merge right away, then. I'll mark it as draft for now.

[holly-cummins]

We probably want some kind of "expected flakes" config on the build reporter.

[gsmet]

Yes. We need to add some infrastructure for that. I'll have a look soon.