Run CI here automatically for every new PR to master branch

[frenzymadness]

Cc: @zmiklank @phracek @pkubatrh

This is what we have here in the settings:

So it should be safe because all collaborators from outside our group will need someone inside the group to approve the CI first every time they open a PR. I will test this because I'm working on another PR and I'll also ask somebody from my team to open an external PR to test the workflow approval process.

[frenzymadness]

It seems to work now. Interesting is that it runs all possible combinations but for some of them, it only checks that the respective Dockerfiles aren't present and skips the test. I'm fine with that. What about you?

[zmiklank]

It seems to work now. Interesting is that it runs all possible combinations but for some of them, it only checks that the respective Dockerfiles aren't present and skips the test. I'm fine with that. What about you?

It seems to me that in this pull_request action the PR status is created by default and not by the Testing Farm as a GitHub Action at it was before. That would explain why the statuses are being created even when the TFaGA is not ran. If that is the case we should disable the statuses in TFaGA with disabling update_pull_request_status input and not setting the pull_request_status_name input.

[zmiklank]

I see one problem in getting to the test logs now.
In the original approach you got to the testing farm results by clicking on "Details" button in the Status. Now this button takes you to the GithubActions tab, where is difficult to find the right URL.

[phracek]

I see one problem in getting to the test logs now.
In the original approach you got to the testing farm results by clicking on "Details" button in the Status. Now this button takes you to the GithubActions tab, where is difficult to find the right URL.

Good catch, this is really mandatory to have logs available under link details. Otherwise, we have to search and memorize what logs have failed and find them in the GitHub Actions tab.

[zmiklank]

New testing farm is released, so that failed tests cause failed action and also the summary is available.
I will trigger the tests again to see, how would it look like here.
[test-all]

Edit: this triggered the other suite that I wanted to trigger.

[zmiklank]

Depends on: sclorg/testing-farm-as-github-action#81

[zmiklank]

If we want to continue with this new approach we need to move this to use the tfaga-wrapper too. There are no more changes planned for the testing GitHub Actions so if we decide to continue with this PR there is a possibility to start with that now (or later of course).
I think there would need to be some changes made in the wrapper itself. I propose to do that on a separate branch in the wrapper's repository.

Lumir what do you think?

[zmiklank]

Moreover, there is still the problem with displaying the result when the TFaGA action is triggered by the pull_request trigger.
The issue exists for that with some examples how we can implement the thing. However it is in the 'nice to have' category :).

[frenzymadness]

I'd like to have that but you know much more than I do about the whole process and configuration. Let me know if there is anything I can do to move this forward. Of course, it's not a top priority.

[zmiklank]

I think we can move the work here: sclorg/testing-farm-as-github-action#139.
And when some safe minimal example of using pull_request / pull_request_target exists in the upstream, we can simply copy it here :).

[phracek]

@frenzymadness Do we really still need this pull request?

[frenzymadness]

The point is to run CI automatically without the need to add a comment. AFAIK you still need to add one of the comments to run the CI, don't you?

[SlouchyButton]

without the need to add a comment

Didn't know this was possible with GitHub features only, instead of using comments. Randomly found this PR now because of the recent activity. I'd very much be for implementing this as constant test comments are irrelevant spam both on the PRs itself and in the inbox.

[frenzymadness]

without the need to add a comment

Didn't know this was possible with GitHub features only, instead of using comments. Randomly found this PR now because of the recent activity. I'd very much be for implementing this as constant test comments are irrelevant spam both on the PRs itself and in the inbox.

That's exactly the reason I'd like to have it implemented. PRs in sclorg are mostly from us so having the ability for the CI to start automatically would be great.