pkg: Performance regression in no-op builds when using dune package management

[shonfeder]

Expected Behavior

Once all dependencies have been installed, there should be virtually no observable perf difference between running dune build in a project with and without use of dune package management.

We agree that ~500ms penalty seems like a generous allowance at the outer limit. So to fix this, if bring perf of null builds using dune package management to within 500ms of using opam for package management, we will solve this.

Actual Behavior

There is a degradation of 3 orders of magnitude in the build perf of no-op builds between these two scenarios:

• A: Using dune pkg to managed dependencies via dune pkg lock
• B: Only using opam to manage dependencies

I.e.:

• In case (A) null dune build take less than 1 second.
• In case (B) null dune build runs can take nearly 13 seconds.

Here's some bench-marking on my old Mac laptop. I am running all the builds on the main branch of mirage/ocaml-cohttp.

Version 3.20.2 using opam deps

The released version

$ opam switch create . # approximate set up
$ dune --version 
3.20.2
$ dune build 
$ hyperfine "dune build"      
Benchmark 1: dune build
  Time (mean ± σ):     850.6 ms ±  30.8 ms    [User: 603.6 ms, System: 221.1 ms]
  Range (min … max):   784.5 ms … 911.6 ms    10 runs

Version 2025-11-06T02:31:54Z with a lock directory

The latest nightly (at the time), using dune pkg

$ rm -rf _opam # and other measures to be sure I am not in a switch managing the deps 
$ dune --version 
"Dune Developer Preview: build 2025-11-06T02:31:54Z, git revision
25157844468bcf5706e1a498fd9eb18e935d87f5"
$ dune pkg lock 
$ dune build                                    
$ hyperfine "dune build"
Benchmark 1: dune build
  Time (mean ± σ):     12.289 s ±  0.107 s    [User: 8.696 s, System: 3.530 s]
  Range (min … max):   12.135 s … 12.474 s    10 runs
$ dune build --verbose # just to show it's not reporting pkg install during the build
Shared cache: enabled
Shared cache location: .../.cache/dune/db
Workspace root: .../mirage/ocaml-cohttp
Auto-detected concurrency: 12
Dune context:
 { name = "default"
 ; kind = lock { default = true }
 ; profile = Dev
 ; merlin = true
 ; fdo_target_exe = None
 ; build_dir = In_build_dir "default"
 ; instrument_with = []
 }
Actual targets:
- alias @@default

Version b202280 using opam deps

$ opam switch create . # approximate setup
$ dune --version
...-gb202280
$ dune build
$ hyperfine --warmup 5 "dune build"       
Benchmark 1: dune build
  Time (mean ± σ):     853.2 ms ±   1.5 ms    [User: 599.4 ms, System: 230.8 ms]
  Range (min … max):   850.3 ms … 855.8 ms    10 runs

Version b202280 using dune pkg

$ rm -rf _opam # and other measures to be sure I am not in a switch managing the deps
$ dune --version
...-gb202280
$ dune pkg lock
$ dune build
$ hyperfine --warmup 5 "dune build"
Benchmark 1: dune build
  Time (mean ± σ):      9.574 s ±  0.061 s    [User: 5.896 s, System: 3.626 s]
  Range (min … max):    9.491 s …  9.685 s    10 runs
$ dune build --verbose 
Shared cache: enabled
Shared cache location: /Users/sf/.cache/dune/db
Workspace root: /Users/sf/Repos/mirage/ocaml-cohttp
Auto-detected concurrency: 12
Dune context:
 { name = "default"
 ; kind = lock { default = true }
 ; profile = Dev
 ; merlin = true
 ; fdo_target_exe = None
 ; build_dir = In_build_dir "default"
 ; instrument_with = []
 }
Actual targets:
- alias @@default

[Alizter]

One of the first steps I would like to take with this is to introduce many more chrome tracing events for both dune pkg lock and dune build with regards to package management. That will help us confirm exactly where the slowdowns are situated and help us guide our optimisations accordingly.