[Junit5 extension] Run test multiple times for the same flag with different values

[aepfli]

Sometimes, you want to ensure that your code behaves the same way for all code paths, even if different feature flags are involved.

Junit5 has parameterized tests to run multiple tests based on a source. This approach is not feasible for us, as it only generates an array of possible argument sets.

We need to find a way (maybe with our annotation) to create such tests. The cartesian test extension of JUnit Pioneer can be an inspiration for this https://github.com/junit-pioneer/junit-pioneer/blob/main/src/main/java/org/junitpioneer/jupiter/cartesian/CartesianTest.java

[aepfli]

theory crafting:

@FlagTest( name = "<flagName>", values = {"true", "false"}) 

[toddbaert]

@UtkarshSharma2612 you might like this.

[UtkarshSharma2612]

Sure let me check this

[UtkarshSharma2612]

Hey @aepfli I understood the requirement a bit, I wanted to know do we want to change the existing testcases or write new testcases for all the providers with this custom annotation.

[aepfli]

It is about adding another Test Annotation, so people who want to write unit tests to test all kinds of variations with one feature flag can use it. maybe some people want to verify, that the code works the same in both cases and is not effected.

eg. currently people would need to write two tests for a boolean to check if the code behaves as expected:

@Test
@Flag(name = "BOOLEAN_FLAG", value = "true")
void trueCase() {
    // your test code
}

@Test
@Flag(name = "BOOLEAN_FLAG", value = "false")
void falseCase() {
    // your test code
}

instead of this they code do

@FlagTest(name = "BOOLEAN_FLAG", values = { "true", "false" })
void bothCases() {
    // this block will run twice, once for 'true' and once for 'false'
}

does this make somewhat sense? - this is only about testing openFeature functionality with junit

[UtkarshSharma2612]

Okay so we want to create a annotation or something that takes two parameters name (string) and values (List) and runs test cases for all the values. Correct me if I am wrong.

[aepfli]

if you do need more guidance or pointers let me know, some people like the challenge of figuring things out on their own, some want more pointers, and i don't want to spoil the fun for you, if you like to dig on your own ;)

[aepfli]

#1091 seems to have set the basics - we could think about a flagsource for the parameterized tests ;)