Repository navigation
Resolve Travis concurrent jobs or consider Azure pipelines #118
Description
https://freenode.logbot.info/saltstack-formulas/20190516#c2180979:
- - - 09:59 myii @gtmanfred A problem is beginning to manifest: as we convert more formulas to semantic-release(and generally getinspectesting working on formulas), our Travis queuing is slowing things down. It only appears to handle 3 jobs at a time; the most I've ever seen is 5, but that may have been on my own forks. So it's not uncommon to see 3/50 active jobs and when things get bad, it can take up to an hour to get results.I'm usually manually cancelling and rerunning (my own) jobs to minimise loads but contributors won't be able to do that. This load will only increase in time, as more formulas get converted. We could start limiting the number of jobs per formula but is there anything else that can be done?10:06 slack1872 except paying for more concurrent builds, I'm not sure... 10:06 myii I reckon we're going to have to start limiting testing per formula. 10:07 myii Unless a formula has specific tests that need to be run on specific combinations of platform/salt-version. 10:08 slack1872 Same problem always pops: you had tests (which is great) and then your tests take too much time...We have the same problem @ work, with 8 kitchen VM created and highstate taking near 15 min. But on Jenkins at least we can run VM in parallel 10:09 myii What do you say to this idea: 10:09 myii 5 instances max, one from each os.10:10 myii 2019.2x2,2018.3x2 and2017.7x1.10:12 myii We've got a nice set to choose from but now this is about how to select which ones to use per formula. 10:12 myii Is 5 sufficient for testing purposes?
https://freenode.logbot.info/saltstack-formulas/20190516#c2181226:
- - - 12:39 gtmanfred no, limit concurrent jobs 12:39 gtmanfred in travis 12:39 gtmanfred you can have as many runs as you want, but it will only run one per repo at a time
https://freenode.logbot.info/saltstack-formulas/20190516#c2181237:
- - - 12:41 gtmanfred if that doesn’t work, we can look into moving to azure pipelines 12:41 gtmanfred i have been playing with that recently and it is pretty good 12:41 myii Excellent. 12:41 gtmanfred not as easy to build a matrix automatically for, but still pretty easy 12:42 myii Look forward to it. 12:43 gtmanfred Here is a version that builds the matrix https://github.com/saltstack/pepper/blob/develop/azure-pipelines.yml 12:43 myii Looks decent. 12:43 gtmanfred and here is a version where you just define it https://github.com/gtmanfred/pepper/blob/f15ce586a02b377ef1f8bdb48e378c5505179074/azure-pipelines.yml 12:44 myii The loop is nicer. 12:44 gtmanfred agreed 12:44 gtmanfred one thing that it does not have is allowed_failures 12:44 gtmanfred and the yaml can be gross 12:45 myii I don't think we're really using that. 12:45 myii So it shouldn't be a problem. 12:45 gtmanfred There is also this limitation with the yaml https://twitter.com/resticbackup/status/1117327446456655873 12:45 gtmanfred it does not parse whole elements in dot net, it parses one line at a time 12:46 myii OK, good to be aware of that. 12:47 myii If this comes to fruition, we'll put these notes in a little doc.
Activity
aboe76 commentedon May 16, 2019
please also test cirrus-ci: https://cirrus-ci.org/#comparison-with-popular-ciaas
myii commentedon May 16, 2019
@aboe76 Would you mind bringing that up in the Slack room? More eyes will see it there. In the meantime, what do you think about limiting the number of runs per formula? Should it be one for each
os; is that sufficient?aboe76 commentedon May 16, 2019
most formula's only needs these different os's, :
when switching to circus ci the concurency limit isn't a factor any more:
https://cirrus-ci.org/#key-highlights
javierbertoli commentedon May 16, 2019
I agree with @aboe76: having a 20+ images available does not mean that we need to test with all of them. I think that to test that a formula works, the rationale should be something like:
tests for the required combination of the above should be added to the test matrix.
I think that part of the review of PRs should check the testing matrix to be enough but not an excess. Testing is easy, knowing what and when to test is not 😋
Also, cirrus-ci sounds interesting 👍
My 2c
myii commentedon May 17, 2019
So after some discussion on Slack prompted by @aboe76, we've got Cirrus CI enabled for the whole org. Some specific notes about BSD:
I've started work on testing Cirrus CI with
cron-formulabut I've not yet had success getting it to play nicely withkitchen. There'll be some special invocation but there aren't any examples out there in the wild that I could use as references.n-rodriguez commentedon May 17, 2019
https://cirrus-ci.org/guide/writing-tasks/#additional-containers
myii commentedon May 18, 2019
So a first example of the difference between 15 and 5, thanks to @daks:
@aboe76 @javierbertoli Are these the 5 images you're suggesting or did you mean stick with the
os_familyimages and only use theosvariants when necessary?By the way, also note that we're leaving all of the platforms available in
kitchen.yml, so that any/all can be tested locally.aboe76 commentedon May 18, 2019
@myii I would prefer to stick with the os_family variants, and then a little bit conservative,
so:
'RedHat' => use centos images
'Debian' => use debian images
'Suse' => use opensuse leap images if possible.
'Bsd' => ??
Having to wait 16 minutes to have a test run for a PR is a bit much...
myii commentedon May 18, 2019
@aboe76 Thanks for that. What do you think of my comment I've just made after you suggested that: saltstack-formulas/ufw-formula#7 (comment)?
aboe76 commentedon May 19, 2019
Yes @myii, I think that is a good comment, like with the testing of the the formula works on OS, the same thing can be applied to salt versions.
The testing should be able to spot regressions and issues earlier, but not in a way that is hindering or slowing everything down. I know about edge cases that we might miss, but I don't think that most of the community will be better served if we stick to the broad road instead of narrow paths.
ci(travis): reduce matrix down to 5 instances (ref: saltstack-formula…
myii commentedon May 19, 2019
So #121 is a suggested workaround to use for the time being. Please have a look so that this selection can be propagated throughout all of the formulas using 10+ instances (i.e. all
semantic-releaseformulas to date).myii commentedon May 19, 2019
I've just noticed that
Limit concurrent jobshas been set to1in Travis. I'm not sure that's such a good idea now, since it reduces the benefit of having these run in parallel, to get releases done quickly. Very busy periods aren't that common, so at quiet times, it really slows things down. I've unset this again for the time being but feel free to discuss this if it is an issue.ci(travis): reduce matrix down to 6 instances (ref: saltstack-formula…
Merge pull request #121 from myii/ci/reduce-travis-matrix-instances
41 remaining items
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform
ci(kitchen+travis): modify matrix to include `develop` platform