Skip to content

WIP: Parallelize evaluation of solvers in bmark_solvers #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
MonssafToukal wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

WIP: Parallelize evaluation of solvers in bmark_solvers #127

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8cd3b44
bmark solvers in parallel
MonssafToukal Aug 25, 2022
e793fd5
bmark solvers in parallel using addprocs
MonssafToukal Sep 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Project.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ BenchmarkProfiles = "ecbce9bc-3e5e-569d-9e29-55181f61f8d0"
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
ColorSchemes = "35d6a980-a343-548e-a6ea-1d62b119f2f4"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
Distributed = "8ba89e20-285c-5b6f-9357-94700520ee1b"
GitHub = "bc5e4493-9b4d-5f90-b8aa-2b2bcaad7a26"
JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819"
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
Expand Down
1 change: 1 addition & 0 deletions src/SolverBenchmark.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
module SolverBenchmark

using Distributed
using Logging
using Printf

Expand Down
24 changes: 19 additions & 5 deletions src/bmark_solvers.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,27 @@ Run a set of solvers on a set of problems.
Any keyword argument accepted by `solve_problems`

#### Return value
A Dict{Symbol, AbstractExecutionStats} of statistics.
A Dict{Symbol, DataFrame} of statistics.
"""
function bmark_solvers(solvers::Dict{Symbol, <:Any}, args...; kwargs...)
function bmark_solvers(solvers::Dict{Symbol, <:Any}, args...;parallel_eval=false, kwargs...)
Copy link
Member

@dpo dpo Sep 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this mandatory? what happens if we try to run the benchmarks in parallel and there is only 1 proc?

Copy link
Contributor Author

@MonssafToukal MonssafToukal Sep 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's mandatory to set parallel_eval=true if you want to benchmark in parallel. If you still have 1 proc, it won't do it in parallel. I added this keyword in case the user is handling multiple procs and does not want evaluate solvers in parallel. We could get rid of it it's not necessary.

Copy link
Member

@dpo dpo Sep 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok we could leave it, at least for debugging, but let's set it to true by default.

stats = Dict{Symbol, DataFrame}()
for (name, solver) in solvers
@debug "running" name solver
stats[name] = solve_problems(solver, args...; kwargs...)
if parallel_eval && length(procs()) > 1
@info "bmark solvers in parallel"
future_stats = Dict{Symbol, Future}()
@sync for (name, solver) in solvers
@async future_stats[name] = @spawnat :any begin
@info "worker $(myid()) running solver $(name)"
solve_problems(solver, args...; kwargs...)
end
end
@sync for (name, future) in future_stats
@async stats[name] = fetch(future)
end
else
for (name, solver) in solvers
@debug "running" name solver
stats[name] = solve_problems(solver, args...; kwargs...)
end
end
return stats
end