Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[backports-release-1.11] Loading backports #56476

Open
wants to merge 10 commits into
base: backports-release-1.11
Choose a base branch
from
Open

[backports-release-1.11] Loading backports #56476

wants to merge 10 commits into from

Conversation

IanButterworth
Copy link
Member

@IanButterworth IanButterworth commented Nov 6, 2024 •
edited
Loading

This is as far as I got with backports to try and get #56329 in to fix #56077

#56329 stumped me.

@vtjnash would you mind helping here.

LilithHafner and others added 7 commits November 6, 2024 13:01
@LilithHafner @IanButterworth
Replace some occurrences of iteration over 1:length with more idiomat…
39e1365 
…ic structures (mostly eachindex) (JuliaLang#55137)

Base should be a model for the ecosystem, and `eachindex(x)` is better
than `1:length(x)` in almost all cases. I've updated many, but certainly
not all examples. This is mostly a NFC, but also fixes JuliaLang#55136.

(cherry picked from commit 0945b9d)
@vtjnash @IanButterworth
add logic to prefer loading modules that are already loaded (JuliaLan…
03d23d0 
…g#55908)

Iterate over the list of existing loaded modules for PkgId whenever
loading a new module for PkgId, so that we will use that existing
build_id content if it otherwise passes the other stale_checks.

(cherry picked from commit 7e2d803)
@KristofferC @vtjnash
remove new references to explicit_loaded_modules
493bec0
@KristofferC @vtjnash
Revert "Extensions: make loading of extensions independent of what pa…
694d00a 
…ckages are in the sysimage (JuliaLang#52841)"

This reverts commit 08d229f.

(cherry picked from commit e3f2f6b)
@vtjnash
fix lookup when extension is in [deps]
d9db8f8 
(cherry picked from commit ad1dc39)
@vtjnash
drop require lock when not needed during loading to allow parallel pr…
7a1aaa9 
…ecompile loading (JuliaLang#56291)

Fixes `_require_search_from_serialized` to first acquire all
start_loading locks (using a deadlock-free batch-locking algorithm)
before doing stalechecks and the rest, so that all the global
computations happen behind the require_lock, then the rest can happen
behind module-specific locks, then (as before) extensions can be loaded
in parallel eventually after `require` returns.

(cherry picked from commit db3d816)
@vtjnash
loading: clean up more concurrency issues (JuliaLang#56329)
501b263 
Guarantee that `__init__` runs before `using` returns. Could be slightly
breaking for people that do crazy things inside `__init__`, but just
don't do that. Since extensions then probably load after `__init__` (or
at least, run their `__init__` after), this is a partial step towards
changing things so that extensions are guaranteed to load if using all
of their triggers before the corresponding `using` returns

Fixes JuliaLang#55556

(cherry picked from commit 9dbdeb4)
@IanButterworth
Copy link
Member Author

@nanosoldier runtests(ALL, vs=":release-1.10")

@IanButterworth
Copy link
Member Author

Whoops, premature pkgeval.. needs JuliaLang/Pkg.jl#4078

@IanButterworth
Copy link
Member Author

Revise tests are hitting ConcurrencyViolationErrors

  | 2024-11-07 08:29:20 EST | ┌ Error: Error during package callback
  | 2024-11-07 08:29:20 EST | │   exception =
  | 2024-11-07 08:29:20 EST | │    1-element ExceptionStack:
  | 2024-11-07 08:29:20 EST | │    ConcurrencyViolationError("lock must be held")
  | 2024-11-07 08:29:20 EST | │    Stacktrace:
  | 2024-11-07 08:29:20 EST | │      [1] concurrency_violation()
  | 2024-11-07 08:29:20 EST | │        @ Base ./condition.jl:8
  | 2024-11-07 08:29:20 EST | │      [2] assert_havelock(l::ReentrantLock, tid::Nothing)
  | 2024-11-07 08:29:20 EST | │        @ Base ./condition.jl:29
  | 2024-11-07 08:29:20 EST | │      [3] assert_havelock
  | 2024-11-07 08:29:20 EST | │        @ ./lock.jl:52 [inlined]
  | 2024-11-07 08:29:20 EST | │      [4] maybe_loaded_precompile(key::Base.PkgId, buildid::UInt128)
  | 2024-11-07 08:29:20 EST | │        @ Base ./loading.jl:2463
  | 2024-11-07 08:29:20 EST | │      [5] parse_pkg_files(id::Base.PkgId)
  | 2024-11-07 08:29:20 EST | │        @ Revise ~/.julia/packages/Revise/8jqWV/src/loading.jl:46
  | 2024-11-07 08:29:20 EST | │      [6] watch_package(id::Base.PkgId)
  | 2024-11-07 08:29:20 EST | │        @ Revise ~/.julia/packages/Revise/8jqWV/src/pkgs.jl:350

https://buildkite.com/julialang/julia-release-1-dot-11/builds/320#019306bf-a5cf-4c37-832f-f9bf7a8e22fd/795-8519

@vtjnash @IanButterworth
add require_lock call to maybe_loaded_precompile (JuliaLang#56027)
e71fdda 
If we expect this to be a public API
(https://github.com/timholy/Revise.jl for some reason is trying to
access this state), we should lock around it for consistency with the
other similar functions.

Needed for timholy/Revise.jl#856

(cherry picked from commit 4cdd864)
@IanButterworth
Copy link
Member Author

That required #56027 which wasn't marked for backport.

If we go with this PR, a few of these need the label added and removed, so it's clear they have been backported.

@IanButterworth
Copy link
Member Author

REPL is detecting unexpected precompiles during startup on this branch that aren't on #56228

  | 2024-11-07 11:00:40 EST | REPL                                            (12) \|        started at 2024-11-07T08:00:40.905
  | 2024-11-07 11:00:43 EST | From worker 12:	┌ Info: REPL: trace compile output:
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Base.var"#1140#1142"{Bool, Symbol, Bool}, Module})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Type{REPL.Terminals.TTYTerminal}, String, Base.TTY, Base.TTY, Base.TTY})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.getproperty), REPL.Terminals.TTYTerminal, Symbol})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Core.memoryref), GenericMemory{:not_atomic, Tuple{String, Int64}, Core.AddrSpace{Core}(0x00)}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Type{REPL.LineEditREPL}, REPL.Terminals.TextTerminal, Bool, Bool})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Core.memoryref), GenericMemory{:not_atomic, Base.Dict{Any, Any}, Core.AddrSpace{Core}(0x00)}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.setproperty!), REPL.Options, Symbol, Bool})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.setproperty!), REPL.LineEditREPL, Symbol, Bool})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Type{REPL.REPLDisplay{Repl} where Repl<:REPL.AbstractREPL}, REPL.LineEditREPL})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.__atreplinit), REPL.LineEditREPL})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Base.var"##s128#268", Vararg{Any, 4}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.:(==)), Bool, Bool})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.hash), String, UInt64})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.:(\|)), UInt128, UInt64})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.run_repl), REPL.AbstractREPL, Any})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Core.kwcall), NamedTuple{names, T} where T<:Tuple where names, typeof(REPL.start_repl_backend), REPL.REPLBackend, Any})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Base.var"#1141#1143", REPL.REPLBackend})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Type{REPL.LineEdit.Modifiers}, Bool})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.sym_in), Symbol, NTuple{12, Symbol}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Type{NamedTuple{(:pass_empty, :suppress_on_semicolon), T} where T<:Tuple}, Tuple{Bool, Bool}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Type{NamedTuple{(:read, :write, :create), T} where T<:Tuple}, Tuple{Bool, Bool, Bool}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Type{Pair{A, B} where B where A}, String, Function})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Type{Pair{A, B} where B where A}, Char, Function})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Type{Pair{A, B} where B where A}, Int64, REPL.LineEdit.KeyAlias})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.getproperty), Pair{String, REPL.LineEdit.var"#54#85"}, Symbol})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Type{Base.Returns{V} where V}, Symbol})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.convert), Type{Function}, Base.Returns{Symbol}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.convert), Type{Any}, Any})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{REPL.var"#75#81"{REPL.LineEditREPL, REPL.REPLBackendRef}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.convert), Type{Array{Symbol, 1}}, Array{UInt8, 1}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.setindex!), Base.Dict{Any, Any}, Any, Int64})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.delete!), Base.Set{Any}, String})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.:(==)), Char, String})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.prepend!), Array{Base.Dict{Any, Any}, 1}, Array{Base.Dict{Any, Any}, 1}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Core.memoryref), GenericMemory{:not_atomic, Base.GenericIOBuffer{GenericMemory{:not_atomic, UInt8, Core.AddrSpace{Core}(0x00)}}, Core.AddrSpace{Core}(0x00)}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Type{REPL.LineEdit.InputAreaState}, Int64, Int64})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.LineEdit.init_state), Any, REPL.LineEdit.Prompt})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{Type{REPL.LineEdit.PromptState}, REPL.Terminals.AbstractTerminal, REPL.LineEdit.Prompt, Base.GenericIOBuffer{GenericMemory{:not_atomic, UInt8, Core.AddrSpace{Core}(0x00)}}, Symbol, Union{Nothing, String}, Array{Base.GenericIOBuffer{GenericMemory{:not_atomic, UInt8, Core.AddrSpace{Core}(0x00)}}, 1}, Int64, REPL.LineEdit.InputAreaState, Int64, Base.Threads.SpinLock, Float64, Float64, Union{Nothing, Base.Timer}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.LineEdit.init_state), Any, REPL.LineEdit.HistoryPrompt})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.LineEdit.init_state), Any, REPL.LineEdit.PrefixHistoryPrompt})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.LineEdit.run_interface), REPL.Terminals.TextTerminal, REPL.LineEdit.ModalInterface, REPL.LineEdit.MIState})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.reseteof), REPL.Terminals.TTYTerminal})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.reseteof), Base.TTY})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.Terminals.raw!), REPL.Terminals.TTYTerminal, Bool})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.Terminals.enable_bracketed_paste), REPL.Terminals.TTYTerminal})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{REPL.LineEdit.var"#281#283"{REPL.LineEdit.MIState, Base.ReentrantLock}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.LineEdit.activate), REPL.LineEdit.ModalInterface, REPL.LineEdit.MIState, REPL.Terminals.AbstractTerminal, REPL.Terminals.TextTerminal})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.LineEdit.activate), REPL.LineEdit.TextInterface, REPL.LineEdit.MIState, REPL.Terminals.AbstractTerminal, REPL.Terminals.TextTerminal})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.setproperty!), REPL.LineEdit.PromptState, Symbol, REPL.LineEdit.InputAreaState})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.LineEdit.refresh_line), Union{Base.GenericIOBuffer{GenericMemory{:not_atomic, UInt8, Core.AddrSpace{Core}(0x00)}}, REPL.LineEdit.MIState, REPL.LineEdit.ModeState}, REPL.Terminals.AbstractTerminal})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.LineEdit.refresh_multi_line), REPL.Terminals.UnixTerminal, Any})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{REPL.LineEdit.var"##refresh_multi_line#20", Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where T<:Tuple{Vararg{Any, N}} where names where N where V, typeof(REPL.LineEdit.refresh_multi_line), REPL.Terminals.UnixTerminal, Any})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.getproperty), Base.MappingRF{Base.ExtremaMap{typeof(Base.identity)}, Base.BottomRF{typeof(Base._extrema_rf)}}, Symbol})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.getproperty), Base.BottomRF{typeof(Base._extrema_rf)}, Symbol})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.iszero), Float64})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.LineEdit.refresh_multi_line), REPL.Terminals.TerminalBuffer, REPL.Terminals.UnixTerminal, Union{REPL.LineEdit.PrefixSearchState, REPL.LineEdit.PromptState}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.Terminals.clear_line), REPL.Terminals.TerminalBuffer})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.Terminals.width), REPL.Terminals.TTYTerminal})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.Terminals.height), REPL.Terminals.TTYTerminal})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.Terminals.hascolor), REPL.Terminals.TTYTerminal})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.LineEdit.write_prompt), REPL.Terminals.AbstractTerminal, REPL.LineEdit.PromptState, Bool})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.write), REPL.Terminals.TerminalBuffer, String})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(REPL.LineEdit.write_prompt), IO, Union{AbstractString, Function}, Bool})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{REPL.var"#103#104"{REPL.LineEditREPL, String}})
  | 2024-11-07 11:00:43 EST | From worker 12:	│ precompile(Tuple{typeof(Base.write), REPL.Terminals.TTYTerminal, Array{UInt8, 1}})
  | 2024-11-07 11:00:43 EST | From worker 12:	└ precompile(Tuple{REPL.LineEdit.var"#282#284"{REPL.Terminals.TTYTerminal, REPL.LineEdit.ModalInterface, REPL.LineEdit.MIState, Base.ReentrantLock, REPL.LineEdit.Prompt}})
  | 2024-11-07 11:00:54 EST | Profile                                         (14) \|    27.54 \|   0.47 \|  1.7 \|    1480.50 \|  2204.61
  | 2024-11-07 11:00:54 EST | Random                                          (14) \|        started at 2024-11-07T08:00:54.990
  | 2024-11-07 11:01:10 EST | REPL                                            (12) \|         failed at 2024-11-07T08:01:10.418
  | 2024-11-07 11:01:10 EST | Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/julia-e71fddabd9/share/julia/stdlib/v1.11/REPL/test/precompilation.jl:40
  | 2024-11-07 11:01:10 EST | Expression: n_precompiles <= expected_precompiles
  | 2024-11-07 11:01:10 EST | Evaluated: 66 <= 0

@nanosoldier
Copy link
Collaborator

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@IanButterworth @nanosoldier @LilithHafner @vtjnash @KristofferC