Multi agent parallel testing in CI #18523
Conversation
✅ No release notes required |
|
This kind of worked, but now Linux is the bottleneck, oh well :) |
|
Linux and MacOs will likely alternate in being the slowest, MacOs also has the smallest pool of available machines. I do like the numbers very much. |
| let addBatchTrait (testCase: ITestCase) = | ||
| let data = Text.Encoding.UTF8.GetBytes testCase.DisplayName | ||
| let hashCode = BitConverter.ToUInt32(sha.ComputeHash(data), 0) | ||
| let batch = hashCode % 4u + 1u |
There was a problem hiding this comment.
just to get a number from 1 to 4. this is hardcoded just for now.
There was a problem hiding this comment.
I wanted to get the batch number from a custom hashCode that is stable between executions. I heard bad things about String.GetHashcode() in this regard.
|
Now it's important to verify the that number of tests add up correctly 😅 |
|
.. number of unique executed tests. I would recommend to download testlog.xml files and have a script count unique test entries and compare two sets (prior exection, new execution from 4 jobs) |
|
So I had some fun splitting the build and test phases into separate jobs. |
|
I verified that in fact all tests do have a batch number trait, and none are left out when run in batches. The failsafe, in case this ever becomes problematic, is to remove the As for the build / test split into separate jobs, I'm not sold on this. it's a matter of reverting the single commit. |
|
For future reference: |
Indeed, the overall time of the CI (and not duration of individual jobs) is the same, if not slower. => I think we can keep each leg do it's own build. |
| if ($testDesktop -and -not $ci ) { | ||
| TestUsingMSBuild -testProject "$RepoRoot\FSharp.sln" -targetFramework $script:desktopTargetFramework -testadapterpath "$ArtifactsDir\bin\FSharp.Compiler.ComponentTests\" | ||
| if ($testDesktop) { | ||
| TestUsingMSBuild -testProject "$RepoRoot\FSharp.sln" -targetFramework $script:desktopTargetFramework |
There was a problem hiding this comment.
I am in general fine with the approach and like it - of testing the entirety of FSharp.sln solution, since for me it is the one I am using the most as well.
Can you please add a sentence or two to the test guide, that any new test project that wishes to be part of CI should be added to this solution?
And inversely, test project which needs special guidance (like editor specific tests) should not be in FSharp.sln - and have to catered for explicitely in the script file, just like it was until now.
There was a problem hiding this comment.
Now I noticed this is not consistent between linux and windows build scripts. The bash script does not run `FSharp.Suite' at all. Probably it could, it runs on net9.0 without problems on Windows.
There was a problem hiding this comment.
I'd leave this for another PR. Bash is not my forte :)
Yes much of this comes from waiting for the windows vms to get running. There is significant wait time between the build and test phases because of this.
I'm away but I'll restore this when I get back 🙂 |
Thanks @majocha . Please mark it as ready for review onace that is changed, I am eager to have this in 👍 |
majocha
force-pushed
the
multi-agent-ci
branch
2 times, most recently
from
ff7fa06 to
88be2a4
Compare
|
This is ready, overall CI run time is now around 48 minutes. It is determined by the slowest |
Make #18517 work.