Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test: add option to record passes and enable it for CI #57690

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

IanButterworth
Copy link
Member

@IanButterworth IanButterworth commented Mar 9, 2025

Related to #57686

Investigating how feasible it would be to record test passes in full so they can be used by buildkite analytics. They are currently just counted.

Todo

  • There's some dummy recording going on that needs removing if this is enabled

    julia/test/runtests.jl

    Lines 381 to 395 in 0b9525b

    elseif isa(resp, Test.TestSetException)
    fake = Test.DefaultTestSet(testname)
    fake.time_end = fake.time_start + duration
    for i in 1:resp.pass
    Test.record(fake, Test.Pass(:test, nothing, nothing, nothing, LineNumberNode(@__LINE__, @__FILE__)))
    end
    for i in 1:resp.broken
    Test.record(fake, Test.Broken(:test, nothing))
    end
    for t in resp.errors_and_fails
    Test.record(fake, t)
    end
    Test.push_testset(fake)
    Test.record(o_ts, fake)
    Test.pop_testset()
  • Figure out why the weakref test is failing. I think the Pass object might be retaining a reference?

@IanButterworth IanButterworth added testsystem The unit testing framework and Test stdlib ci Continuous integration labels Mar 9, 2025
@IanButterworth IanButterworth mentioned this pull request Mar 9, 2025
6 tasks
@@ -1101,7 +1105,11 @@ struct FailFastError <: Exception end
# For a broken result, simply store the result
record(ts::DefaultTestSet, t::Broken) = (push!(ts.results, t); t)
# For a passed result, do not store the result since it uses a lot of memory
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# For a passed result, do not store the result since it uses a lot of memory

This comment should be removed/changed to mention that this only happens conditionally with this PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Yeah this is more of an investigation of the performance/test side effect implications currently.

Test.TESTSET_PRINT_ENABLE[] = false
ENV["JULIA_TEST_RECORD_PASSES"] = Base.get_bool_env("CI", false)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@KristofferC
Copy link
Member

For some history: #20137 + #20150

@DilumAluthge
Copy link
Member

Instead of keeping all the successful tests in memory, could we stream them to a file?

This would be off by default, and we'd turn it on only in Buildkite CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Continuous integration testsystem The unit testing framework and Test stdlib
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants