Enhance the compatibility test kit to support testing of multiple versions of connector

[i322926sap]

Description

Current setup will only test a snapshot version of connector against one stable version of connector.
The EDC compatibility test suite should be enhanced to support testing against a matrix of connector versions. This will allow us to systematically verify compatibility between different EDC connector releases, ensuring interoperability and identifying potential issues arising from version mismatches. This should include both forward and backward compatibility testing.

Acceptance Criteria

• Test against multiple versions of connector
• Test agains multiple protocol versions

Additional Information

Currently there is only one test class where all the configurations are hardcoded. This also includes the combinations of connector version and protocol versions. With this setup the test landscape will not scale as we will more versions coming up in the future and to ensure the compatibility among all the versions we need to have a matrix combination setup in the test.

[lgblaumeiser]

Hi @i322926sap , the intention to support multiple versions is absolutely clear and this was only the start to test against the currently only relevant version 0.7.7. The main focus here is basically to enhance the test cases as they are extremely rudimentary at the moment.

Another issue, we currently focus on, is to bring together the EDC E2E tests in https://github.com/eclipse-tractusx/tractusx-edc/tree/main/edc-tests with the compatibility tests, as the e2e tests are basically a special case of the compatibility tests targeted at a test of an EDC version against itself. Expect work on this in the near future. This will improve the compatibility tests a lot, as the e2e tests are further in test depth.

In general, I am not sure, if we need more versions to test. The main target of the current setup is, to be able to test the actively developed version against the last relevant. As fixed versions are either compatible or not, continuous testing of the fact does not add any value, right?

The idea is that if the currently developed version is tested concerning compatibility to the last relevant version, the compatibility to previous version could be deduced transitively. If the about to be released version is compatible to a version that has been tested to a previous version, the compatibility to that previous version can be deduced as long as no substantial improvements in the test cases have been achieved.

[i322926sap]

Hi @lgblaumeiser Thank you for the feedback.
I discussed with @wolf4ood and this was one of the suggestions provided by him.
@wolf4ood would you like to add more thoughts on this ?

[wolf4ood]

Hi @lgblaumeiser @i322926sap

yes I suggested that this repo is more suitable for a matrix testing compared to the upstream one.
In EDC upstream we only supports the main branch and the last stable, which differs from the tractsux-edc which more versions need to be supported.

We bootstrapped this project based on eclipse-tractusx/sig-release#908 with the initial code taken the upstream EDC compatibility tests but with tractusx-edc specific feature.

One of the requirement was to test multiple versions of EDC and back then I suggested to @i322926sap that having a flexible tests suite and more tests could be an improvement to compatibility checks.

I was not aware of the future planning on this repo or improving the EDC E2E tests since I'm currently not on the project and in that case @i322926sap I would suggest to align with @lgblaumeiser how you better help in this improvements

Thanks

[lgblaumeiser]

Hi @i322926sap , @wolf4ood

we started with the requirement roadmap issue from @DanielaWuensch that showed a whole matrix and when I looked at your first approach, Enrico, I objected but when I thought about it, you were right, it only makes sense to test a current development stream against older versions. This is true for the main development but also for bugfix branches which are not well supported as of now. The point is, that if a test shows for two existing releases, that they are compatible, they stay compatible and the other way around. If at some point, we have such an improvement in the testing, that we want to compare two existing releases, it is not a big thing, to start a branch from one of the release tags and run the build on that branch.

So the question is, what are the key definiciencies I see as of now:

• As far as I understand, the current implementation does not support easily bugfix branches, that would be something important.
• The amount of testing is really a bare minimum, it should be a main focus to improve the executed test cases, and it does not make sense to maintain two sets of test cases, e2e tests for testing the same version against each other and compatibility tests that test an older version against the current one.
• It could make sense to provide an option to specify a certain old version for a manually triggered compatibility test for special cases of interest.

Currently, there is only one older version relevant, that is version 0.7.7, which is the current counterpart. We deliberately decided against updating that to 0.8.0 as the dataspace participants mostly use the 0.7.7 version as of now. If we get 0.9.0 with Io into the operated dataspace, we can discuss again, but I still think that running the tests continuously against the latest relevant version should be enough, as this should already have proven its compatibility with the previous versions, so what else should the tests find out in addition to the result of the test against the previous version?

[suh-rao]

@lgblaumeiser Do you have any information regarding when decision on Io will be made?

[lgblaumeiser]

HI @suh-rao , what do you mean by decision on Io?

[suh-rao]

Hi @lgblaumeiser In previous comment you mentioned that << If we get 0.9.0 with Io into the operated dataspace, we can discuss again,>>. I was having discussion with Ulf about same and wanted to know how we should take this 'issue' about compatibility forward.

[lgblaumeiser]

Hi @suh-rao, the if was a when, bad englisch, sorry. Version 0.9.0 will be released with 25/03 and the version is one targeted for use in an ecosystem release. We currently know, that 0.8.1 is a version used by some participants as well. So testing the development after 0.9.0 against 0.9.0 and 0.7.7 and potentially 0.8.1 is something that can be discussed.

But with the current setup, it does not really make sense as the current testcase is pretty straight forward, and it has been succesful tested with 0.7.7 as counterpart. We will only burn CO2, if we test against more versions because it would be a strange surprise when we find an inconsistency against one of the versions only.

That is why I prioritize the activities targeted in the coming Tractus-X development phase which is centered around the idea to bring the e2e and the compatibility tests closer together. This will already add a multi-version test as this requires that the current development is tested against the current development and one older version. This way, we get the multi-version support for free and we can, based on a solid discussion, add more versions to test against.

The approach shown in the open PR, btw, is not target oriented, as it adds a specific setup for the version 0.7.6. The general mechanism must be more flexible, e.g., defining the versions and branches by setting Github action parameters or using a matrix approach in the action.