feat: tweak weekly tests #1934
Merged
feat: tweak weekly tests #1934
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DavePearce
force-pushed
the
1932-feat-tweak-weekly-tests
branch
28 times, most recently
from
2630105 to
ef72a69
Compare
DavePearce
force-pushed
the
1932-feat-tweak-weekly-tests
branch
from
ef72a69 to
36c65d9
Compare
DavePearce
force-pushed
the
1932-feat-tweak-weekly-tests
branch
2 times, most recently
from
79aa6ae to
dbb8a1d
Compare
amkCha
reviewed
Mar 25, 2025
amkCha
reviewed
Mar 25, 2025
DavePearce
force-pushed
the
1932-feat-tweak-weekly-tests
branch
2 times, most recently
from
1793872 to
7b5ded0
Compare
Since the precompile call tests have a different dynamic from other weekly tests, have moved all of them into the group that was previously the weekly ecpairing suite. That way, the parallelism for this group can be dramatically increased without causing other (costly and unrelated) weekly tests from failing.
The purpose of this is just for testing to let me run the workflow prior to it being merged in (since it hasn't been run before).
The challenge with using a high level of parallelism is that we end up creating a large number of JVMs. With a minimum heap size of 4GB, we completely max out memory at 32 processes. That's ignoring memory required for the OS and/or go-corset itself. However, most of these short-lived tests do not need anything like 4GB to complete. Setting a minimum heap size of 512MB means the average test consumes much less RAM, whilst keeping the maximum heap size at 8GB means there is room for them to expand as needed. The final challenge with increasing the degree of parallelism is go-corset itself. This also uses a large number of cores, even for a single test. Therefore, there is a careful balance needed to make effective use of all available cores.
This enables random sampling using the day of the month as a seed for the RNG. This means that, in principle, we can recreate any failures by looking up on which day the test failed. At the same time, we benefit from wider testing over a period of time. A default sample size is given individually for each test, and this can be overridden using a single environment variable PRC_CALLTESTS_SAMPLE_SIZE. Hopefully, this provides the level of flexibility needed.
This consolidates all of the prcCallTests into the existing weeklyTests group. This makes sense as they all have roughly similar test profiles. The weeklyTests now has a much more aggressive degree of parallelism. In addition, this sets more suitable default sample sizes for the PRC calltests. Specifically, defaults are set to approx 10% of the overall parameter space.
DavePearce
force-pushed
the
1932-feat-tweak-weekly-tests
branch
2 times, most recently
from
3d3b7a6 to
f97b137
Compare
amkCha
approved these changes
Mar 26, 2025
This splits the weekly tests up into two distinct jobs so they can run in parallel. This hopefully should allow for a larger sampling size.
DavePearce
force-pushed
the
1932-feat-tweak-weekly-tests
branch
from
f97b137 to
757b3f8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ready to merge.