-
-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test: add option to record passes and enable it for CI #57690
base: master
Are you sure you want to change the base?
Test: add option to record passes and enable it for CI #57690
Conversation
@@ -1101,7 +1105,11 @@ struct FailFastError <: Exception end | |||
# For a broken result, simply store the result | |||
record(ts::DefaultTestSet, t::Broken) = (push!(ts.results, t); t) | |||
# For a passed result, do not store the result since it uses a lot of memory |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# For a passed result, do not store the result since it uses a lot of memory |
This comment should be removed/changed to mention that this only happens conditionally with this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Yeah this is more of an investigation of the performance/test side effect implications currently.
Test.TESTSET_PRINT_ENABLE[] = false | ||
ENV["JULIA_TEST_RECORD_PASSES"] = Base.get_bool_env("CI", false) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of keeping all the successful tests in memory, could we stream them to a file? This would be off by default, and we'd turn it on only in Buildkite CI. |
Related to #57686
Investigating how feasible it would be to record test passes in full so they can be used by buildkite analytics. They are currently just counted.
Todo
julia/test/runtests.jl
Lines 381 to 395 in 0b9525b