Tentatively slated for 5.3 RC1 for team discussion

@johnchurchill Could you please provide a few more details about your use case, particularly how you've written your tests? I assume it involves base classes that inherit @ParameterizedTest methods?

I have also been trying to envision what you are trying to do and was not able.

My best guess is that you have some sort of ArgumentsProvider that returns no elements, but still I am not sure. If my guess is valid, this behaviour was explicitly added as a solution for #923.

It would be great if you could provide a skeleton test that exhibits your issue.

Yes, there is a superclass of all tests which does the parameter loading and execution. There are thousands of subclasses that implement all kinds of functionality. Each (non-test) class is the logic behind an interactive web page in the system. Here is a boiled-down version of that superclass:

@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = SpringDevTestServerConfiguration.class)
@TestExecutionListeners(listeners = { DependencyInjectionTestExecutionListener.class })
@ComponentScan(lazyInit = true)
@TestInstance(PER_CLASS)
public abstract class AbstractConfigTestClass implements ApplicationContextAware {

	protected List<SingleConfigTestDescriptor> configTests() {
		String testClassName = this.getClass().getName();
		// remove "Test" on the end to obtain the config class
		String clzName = testClassName.substring(0, testClassName.length() - 4);
		try {
			Class configClz = Class.forName(clzName);
			List<SingleConfigTestDescriptor> params = ConfigToTestParams.getTestParameters(configClz);
			if (params.size() == 0) {
				params.add(null); // need to add null here to avoid junit 5 failing
			}
			return params;
		}
		catch (ClassNotFoundException e1) {
			throw new RuntimeException("Could not find expected class: " + clzName, e1);
		}
	}

	@ParameterizedTest
	@MethodSource("configTests")
	public void runTest(SingleConfigTestDescriptor test) {
		if (test == null) {
			return; // nothing to do; this is the null to avoid the junit 5 failure
		}
		// use the test data to run the test against the class
	}

}

By the time we know that there are no available tests to run, it's too late.. JUnit 5 will throw an exception. There are 6 lines in there whose only purpose is to make it run successfully like it did under JUnit 4. I just started working with JUnit 5 a couple of days ago, so I haven't had the chance to look into using ArgumentsProvider. Would using ArgumentsProvider avoid the failure?

In any case, we should improve the error message.

"Configuration error: You must provide at least one argument for this @ParameterizedTest" --> "Configuration error: You must provide at least one invocation context for this @ParameterizedTest".

The difference being "argument" vs. "invocation context".

And the invocation context can still supply zero "arguments" to the method, though that's not the topic of this issue.

FYI: I reworded the title of this issue accordingly.

Keep in mind that a @TestTemplate method in general cannot currently have zero invocations.

In other words, see org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.validateWasAtLeastInvokedOnce(int). 😉

I took a look at TestTemplateTestDescriptor, and yes the zero invocations would be stopped there too. I tried taking both of those preconditions out (TestTemplateTestDescriptor and ParameterizedTestExtension) and making a simple class with one @Test and one @ParameterizedTest which refers to a @MethodSource which returns an empty list. The result is that 1/1 tests pass (just bar2()), which is the same behavior we had in JUnit 4. There is no side-effect of allowing the @ParameterizedTest to run with no invocations.

public class FooTest {

	@ParameterizedTest
	@MethodSource("data")
	void bar1(String s) {
	}

	@Test
	public void bar2() {
		// pass
	}

	public static List<String> data() {
		// pretend we have no test cases in the QA database
		return new ArrayList<>();
	}
}

There is no side-effect of allowing the @ParameterizedTest to run with no invocations.

Well, that's only partially true.

It's a breaking change in the sense that configuration errors previously caused the parameterized test container to fail; whereas, now such configuration errors would be silently ignored.

Of course, your use case is not a "configuration error". So we have to decide how to best support both scenarios. 😉

I suppose, if we wanted to support zero invocations, we could simply generate a log message informing the user. The tricky part is what log level to choose. Is that DEBUG, INFO, or WARN?

The result is that 1/1 tests pass

In such cases, I don't think the parameterized test container should be marked as "successful" (or somehow ignored, whatever was the case when you implemented that PoC).

Rather, I think we should mark the parameterized test container as "skipped" and provide a "reason" why it was skipped.

@junit-team/junit-lambda, thoughts?

As a side note, since I'm the author of the Spring TestContext Framework...

@johnchurchill, out of curiosity, why do you disable all Spring test listeners except DependencyInjectionTestExecutionListener?

That's enabled by default.

Were other listeners somehow causing problems?

This case reminds me of the similar case #1298 which was handled by the --fail-if-no-tests cl option.

The similarity is that they both exhibit the same core dilemma:
If I have zero elements to process, is that because the user has done some error or is this lack of items actually valid and intentional? The error could be in configuration as in both cases or programmatic in the case of providing arguments_(invocation contexts)_.

The difference between the two is the default is reversed.

• In the case of the Console Launcher. In cl the default was not to fail and we added an option to change this behavior.
• For the parameterized case we fail if zero invocation contexts.

Maybe we could take a similar approach also in the case- with some sort of flag or parameter eg on the annotation.

For me personally I think the current default to fail if no elements are present and in esense guarding against error is important. I seen many case where something was silently turned off due to error and the error was missed.
*Also adding a skipped note in the execution plan could be in many cases be effectively silent if it does not actually fail a test run

n any case, we should improve the error message.

"Configuration error: You must provide at least one argument for this @ParameterizedTest" --> "Configuration error: You must provide at least one invocation context for this @ParameterizedTest".

The difference being "argument" vs. "invocation context".

I don't think invocation context would be more meaningful for the normal user. Though indeed it is more meaningful for someone who understands the internal of the platform.

If you consult the relevant section in the User Guide about ParameterizedTests there is no mention, let alone an explanation, of "Invocation Context".

Sure.... "invocation context" is very technical.

How about the following instead?

"Configuration error: You must configure at least one set of arguments for this @ParameterizedTest"

Good points, @gaganis.

Maybe we could take a similar approach also in the case- with some sort of flag or parameter eg on the annotation.

I tend to agree that adding an opt-in flag as a annotation attribute in @ParameterizedTest would be a better, more robust, and more user friendly solution.

How about the following?

@ParameterizedTest(failIfNoArgumentsProvided = false)

... with the default being true.

Though, failIfNoArgumentsProvided is a bit verbose. Perhaps we can come up with something more succinct.

Maybe the following is better:

@ParameterizedTest(requireArguments = false)

How about the following instead?

"Configuration error: You must configure at least one set of arguments for this @ParameterizedTest"

That is exactly was I was thinking now too :)

@johnchurchill, would the following proposal suit your needs?

@ParameterizedTest(requireArguments = false)

FYI: I improved the error message in b1e533c.

Yes, requireArguments = false would suit my needs perfectly. I hate to see more corner-case parameters being introduced though. The simple API is what makes JUnit so approachable.

Yes, requireArguments = false would suit my needs perfectly.

Thanks for the feedback.

I hate to see more corner-case parameters being introduced though. The simple API is what makes JUnit so approachable.

Well, therein lies the rub: We can't have our cake and eat it, too.

As already pointed out, switching the behavior would no longer be user friendly regarding user configuration errors. So, IMHO, the only viable option is to make it an opt-in feature while retaining backwards compatibility and upholding UX.

Of course, if anyone has any better ideas, I'm all ears.

Thanks for all of the comments, everyone!

We will discuss this topic again within the team.

Regarding the comment

It's a breaking change in the sense that configuration errors previously caused the parameterized test container to fail; whereas, now such configuration errors would be silently ignored.

I can hardly imagine such a configuration error, can you name one?

I can hardly imagine such a configuration error, can you name one?

Yes.

@ParameterizedTest
@MethodSource
void strings(String str) {
}

static List<String> strings() {
	return List.of();
}

That's an obvious user/configuration error.

Team Decision

• Allow a TestTemplateInvocationContextProvider to signal that it may return an empty Stream from provideTestTemplateInvocationContexts() -- likely via a new default method which returns false and can be overridden in concrete implementations.
• Provide an attribute in @ParameterizedTest which allows the user to signal that zero invocations are permissible -- for example, requireArguments = false (where requireArguments defaults to true) as mentioned previously in the discussions in this issue.

We would also benefit from this fix. Is any help required here?

@nskvortsov A PR implementing the above team decision would be welcome!

Here it is 🙂,
Hope it helps.

Can I help in any way to proceed with code review of the pull request?

No, I don't think so. It's just a matter of someone from the core team taking a proper look.

Dear team, I would be happy to improve my pull request. Please give me some feedback on it )

@nskvortsov Sorry for the delay! I have reviewed the PR just now.