Resolve Travis concurrent jobs or consider Azure pipelines

[myii]

https://freenode.logbot.info/saltstack-formulas/20190516#c2180979:

-	-	-
09:59	myii	@​gtmanfred A problem is beginning to manifest: as we convert more formulas to semantic-release (and generally get inspec testing working on formulas), our Travis queuing is slowing things down. It only appears to handle 3 jobs at a time; the most I've ever seen is 5, but that may have been on my own forks. So it's not uncommon to see 3/50 active jobs and when things get bad, it can take up to an hour to get results.I'm usually manually cancelling and rerunning (my own) jobs to minimise loads but contributors won't be able to do that. This load will only increase in time, as more formulas get converted. We could start limiting the number of jobs per formula but is there anything else that can be done?
10:06	slack1872	except paying for more concurrent builds, I'm not sure...
10:06	myii	I reckon we're going to have to start limiting testing per formula.
10:07	myii	Unless a formula has specific tests that need to be run on specific combinations of platform/salt-version.
10:08	slack1872	Same problem always pops: you had tests (which is great) and then your tests take too much time...We have the same problem @ work, with 8 kitchen VM created and highstate taking near 15 min. But on Jenkins at least we can run VM in parallel
10:09	myii	What do you say to this idea:
10:09	myii	5 instances max, one from each os.
10:10	myii	2019.2 x2, 2018.3 x2 and 2017.7 x1.
10:12	myii	We've got a nice set to choose from but now this is about how to select which ones to use per formula.
10:12	myii	Is 5 sufficient for testing purposes?

https://freenode.logbot.info/saltstack-formulas/20190516#c2181226:

-	-	-
12:39	gtmanfred	no, limit concurrent jobs
12:39	gtmanfred	in travis
12:39	gtmanfred	you can have as many runs as you want, but it will only run one per repo at a time

https://freenode.logbot.info/saltstack-formulas/20190516#c2181237:

-	-	-
12:41	gtmanfred	if that doesn’t work, we can look into moving to azure pipelines
12:41	gtmanfred	i have been playing with that recently and it is pretty good
12:41	myii	Excellent.
12:41	gtmanfred	not as easy to build a matrix automatically for, but still pretty easy
12:42	myii	Look forward to it.
12:43	gtmanfred	Here is a version that builds the matrix https://github.com/saltstack/pepper/blob/develop/azure-pipelines.yml
12:43	myii	Looks decent.
12:43	gtmanfred	and here is a version where you just define it https://github.com/gtmanfred/pepper/blob/f15ce586a02b377ef1f8bdb48e378c5505179074/azure-pipelines.yml
12:44	myii	The loop is nicer.
12:44	gtmanfred	agreed
12:44	gtmanfred	one thing that it does not have is allowed_failures
12:44	gtmanfred	and the yaml can be gross
12:45	myii	I don't think we're really using that.
12:45	myii	So it shouldn't be a problem.
12:45	gtmanfred	There is also this limitation with the yaml https://twitter.com/resticbackup/status/1117327446456655873
12:45	gtmanfred	it does not parse whole elements in dot net, it parses one line at a time
12:46	myii	OK, good to be aware of that.
12:47	myii	If this comes to fruition, we'll put these notes in a little doc.

[aboe76]

please also test cirrus-ci: https://cirrus-ci.org/#comparison-with-popular-ciaas

[myii]

@aboe76 Would you mind bringing that up in the Slack room? More eyes will see it there. In the meantime, what do you think about limiting the number of runs per formula? Should it be one for each os; is that sufficient?

[aboe76]

most formula's only needs these different os's, :

• debian (this covers ubuntu/debian derivites)
• centos (this covers redhat/fedora derivites)
• opensuse (because sles/susemanager stuff)
• bsd

when switching to circus ci the concurency limit isn't a factor any more:
https://cirrus-ci.org/#key-highlights

• free for opensource
• no concurency limit
• flexible execution
• matrix builds

[javierbertoli]

most formula's only needs these different os's, :
* debian (this covers ubuntu/debian derivites)
* centos (this covers redhat/fedora derivites)
* opensuse (because sles/susemanager stuff)
* bsd

I agree with @aboe76: having a 20+ images available does not mean that we need to test with all of them. I think that to test that a formula works, the rationale should be something like:

1. Latest Salt + py3 on

• debian
• centos
• opensuse
• bsd

2. If something is known to fail or behaves different with

• a particular distro (ie, fedora),
• distro release (ie, centos-6)
• salt version (ie, 2018)
• python version (ie, py )

tests for the required combination of the above should be added to the test matrix.

I think that part of the review of PRs should check the testing matrix to be enough but not an excess. Testing is easy, knowing what and when to test is not 😋

Also, cirrus-ci sounds interesting 👍

My 2c

[myii]

So after some discussion on Slack prompted by @aboe76, we've got Cirrus CI enabled for the whole org. Some specific notes about BSD:

-	-	-
21:07	gtmanfred	you would need to use vagrant for bsd
21:08	gtmanfred	realistically though, you could use kitchen passthrough, to run the commands on the bsd instances, instead of creating a new docker container to run the tests on
21:10	gtmanfred	when you get to it, see how we using the proxy for windows on appveyor https://github.com/saltstack/salt-jenkins/blob/develop/.kitchen.appveyor.yml
21:10	gtmanfred	it should translate to bsd probably

I've started work on testing Cirrus CI with cron-formula but I've not yet had success getting it to play nicely with kitchen. There'll be some special invocation but there aren't any examples out there in the wild that I could use as references.

[n-rodriguez]

I've started work on testing Cirrus CI with cron-formula but I've not yet had success getting it to play nicely with kitchen. There'll be some special invocation but there aren't any examples out there in the wild that I could use as references.

https://cirrus-ci.org/guide/writing-tasks/#additional-containers

[myii]

So a first example of the difference between 15 and 5, thanks to @daks:

Instances	Running time	Total time
15	16 min 24 sec	37 min 20 sec
5	7 min 17 sec	10 min 47 sec

@aboe76 @javierbertoli Are these the 5 images you're suggesting or did you mean stick with the os_family images and only use the os variants when necessary?

By the way, also note that we're leaving all of the platforms available in kitchen.yml, so that any/all can be tested locally.

[aboe76]

@myii I would prefer to stick with the os_family variants, and then a little bit conservative,
so:
'RedHat' => use centos images
'Debian' => use debian images
'Suse' => use opensuse leap images if possible.
'Bsd' => ??

Having to wait 16 minutes to have a test run for a PR is a bit much...

[myii]

@aboe76 Thanks for that. What do you think of my comment I've just made after you suggested that: saltstack-formulas/ufw-formula#7 (comment)?

[aboe76]

Yes @myii, I think that is a good comment, like with the testing of the the formula works on OS, the same thing can be applied to salt versions.

The testing should be able to spot regressions and issues earlier, but not in a way that is hindering or slowing everything down. I know about edge cases that we might miss, but I don't think that most of the community will be better served if we stick to the broad road instead of narrow paths.

[myii]

So #121 is a suggested workaround to use for the time being. Please have a look so that this selection can be propagated throughout all of the formulas using 10+ instances (i.e. all semantic-release formulas to date).