From ff00b583e4005300f674a3977d6025d12791c67c Mon Sep 17 00:00:00 2001 From: Marek Kubica Date: Wed, 3 Dec 2025 17:29:49 +0100 Subject: [PATCH] Update documentation to use autolocking Signed-off-by: Marek Kubica --- .../dune-package-management/dependencies.md | 123 +++++++++++------- .../dependencies/dune-workspace | 2 + .../dune-package-management/index.md | 1 + .../dune-package-management/locking.md | 42 ++++++ .../dune-package-management/pinning.md | 39 +++--- .../dune-package-management/repos.md | 48 ++++--- .../repos/dune-workspace | 4 +- .../dune-package-management/setup.md | 60 +++++---- .../setup/dune-workspace | 2 + 9 files changed, 217 insertions(+), 104 deletions(-) create mode 100644 doc/tutorials/dune-package-management/dependencies/dune-workspace create mode 100644 doc/tutorials/dune-package-management/locking.md create mode 100644 doc/tutorials/dune-package-management/setup/dune-workspace diff --git a/doc/tutorials/dune-package-management/dependencies.md b/doc/tutorials/dune-package-management/dependencies.md index 30f5e2d6f14..25ee054aec8 100644 --- a/doc/tutorials/dune-package-management/dependencies.md +++ b/doc/tutorials/dune-package-management/dependencies.md @@ -9,7 +9,8 @@ Much like in regular projects, to add a library we need to add a dependency to it. For simplicity we will use the popular `fmt` library as an example, but any package from the [package repository](https://ocaml.org/packages) can be used. -First we update the `dune-project` file to add a dependency on the opam package. +To do so we update the `dune-project` file to add a dependency on the opam +package. ::::{dropdown} `dune-project` :icon: file-code @@ -19,33 +20,27 @@ First we update the `dune-project` file to add a dependency on the opam package. :emphasize-lines: 8 ::: +Here we define the OPAM packages that we want to use, along with the version +constraints these dependencies should adhere to. + :::: -After this change to our project dependencies, we need to relock dependencies -to update our lock directory with the new packages. +::::{dropdown} `dune-workspace` +:icon: file-code -``` -$ dune pkg lock -Solution for dune.lock: -- base-unix.base -- fmt.0.9.0 -- ocaml.5.2.0 -- ocaml-base-compiler.5.2.0 -- ocaml-config.3 -- ocamlbuild.0.15.0+dune -- ocamlfind.1.9.6+dune -- topkg.1.0.7 -``` +:::{literalinclude} dependencies/dune-workspace +:language: dune +:emphasize-lines: 2 +::: -You can see a lot of new dependencies, among these `fmt`. +In this file we direct Dune to enable package management in the current +workspace. The `pkg` stanza configures Dune to manage the declared dependencies +automatically. -:::{note} -The list of packages being output includes all dependencies of your project, -including transitive dependencies. -::: +:::: -This will take care of installing the dependencies, but we still need to add it to -our build as a library as usual: +This configuration will take care of installing the dependencies, but we still +need to add it to our build as a library as usual: ::::{dropdown} `dune` :icon: file-code @@ -55,11 +50,12 @@ our build as a library as usual: :emphasize-lines: 3 ::: -Adding a library dependency to our `dune` file via the `libraries` stanza. +Adding a library dependency to our `dune` file via the `libraries` stanza. This +is unchanged from the usual Dune workflow. :::: -This will allow us to use the `Fmt` module in our OCaml code. +This change will allow us to use the `Fmt` module in our OCaml code. ::::{dropdown} `test.ml` :icon: file-code @@ -80,8 +76,34 @@ To build it we just call `build` again. $ dune build ``` -which will download and install the new dependencies and build our project as -before. +Dune will notice that the project depends on new packages. Thus it will re-run +the internal dependency solver to find a solution for the set of packages to +use. It will then use this new solution to download, build and install these +dependencies automatically. + +We can check the build log in `_build/log` and see the packages that the Dune +solver has selected: + +``` +... +# Dependency solution for +# _build/.sandbox//_private/default/.lock/dune.lock: +# - base-unix.base +# - fmt.0.11.0 +# - ocaml.5.4.0 +# - ocaml-base-compiler.5.4.0 +# - ocaml-compiler.5.4.0 +# - ocaml-config.3 +# - ocamlbuild.0.16.1+dune +# - ocamlfind.1.9.8+dune +# - topkg.1.1.1 +... +``` + +:::{note} +The list of packages being output includes all dependencies of your project, +including transitive dependencies. +::: As we see, the code works and uses `fmt` to do the pretty-printing: @@ -106,35 +128,42 @@ used for opam dependencies in the `dune-project` file. :::: -This ensures the `fmt` package to install will be compatible with -our request. These constraints will be taken into account the next time the -package is locked: +This change ensures the `fmt` package to install will be compatible with our +request. These constraints will be taken into account the next time the build +system is ran. -``` -$ dune pkg lock -Solution for dune.lock: -- base-unix.base -- fmt.0.9.0 -- ocaml.5.2.0 -- ocaml-base-compiler.5.2.0 -- ocaml-config.3 -- ocamlbuild.0.15.0+dune -- ocamlfind.1.9.6+dune -- topkg.1.0.7 +```sh +dune build ``` -The version of `fmt` picked is indeed between `0.6` and `1.0`. +Checking `_build/log` again reveals that our change was taken into account: + +``` +... +# Dependency solution for +# _build/.sandbox//_private/default/.lock/dune.lock: +# - base-unix.base +# - fmt.0.9.0 +# - ocaml.5.4.0 +# - ocaml-base-compiler.5.4.0 +# - ocaml-compiler.5.4.0 +# - ocaml-config.3 +# - ocamlbuild.0.16.1+dune +# - ocamlfind.1.9.8+dune +# - topkg.1.1.1 +... +``` ## Removing Dependencies Given all dependencies are defined in the `dune-project` file, removing a -dependency means to remove the dependency from the `depends` field of your -`dune-project` and relocking the project. +dependency just means to remove the dependency from the `depends` field of your +`dune-project`. -The new lock directory will not depend on the package anymore, and in future -builds, the package will not be accessible as `library` anymore. +From then on the project will not depend on the package anymore, and in future +builds the package will not be accessible as `library` anymore. :::{note} -The removed dependency might still be part of the lock directory if some other -dependency of your project depends on it. +The removed dependency might still be accessible if some other dependency of +your project depends on it, thus if it is a transitive dependency. ::: diff --git a/doc/tutorials/dune-package-management/dependencies/dune-workspace b/doc/tutorials/dune-package-management/dependencies/dune-workspace new file mode 100644 index 00000000000..3562360fe0d --- /dev/null +++ b/doc/tutorials/dune-package-management/dependencies/dune-workspace @@ -0,0 +1,2 @@ +(lang dune 3.21) +(pkg enabled) diff --git a/doc/tutorials/dune-package-management/index.md b/doc/tutorials/dune-package-management/index.md index 9dc6069ea56..430e4af61e7 100644 --- a/doc/tutorials/dune-package-management/index.md +++ b/doc/tutorials/dune-package-management/index.md @@ -24,4 +24,5 @@ setup dependencies pinning repos +locking ::: diff --git a/doc/tutorials/dune-package-management/locking.md b/doc/tutorials/dune-package-management/locking.md new file mode 100644 index 00000000000..81b216d1dfa --- /dev/null +++ b/doc/tutorials/dune-package-management/locking.md @@ -0,0 +1,42 @@ +# Locking Your Dependencies + +In the default use-case Dune will automatically determine which packages to +install, by reading the package constrains, determining compatible versions and +installing the dependencies automatically. + +For many projects this is a good and acceptable behavior as users often want to +use new versions of their dependencies. However some projects might want to +keep a fixed set of (transitive) dependencies that is only updated manually. + +## Create a lock directory manually + +If a lock directory exists in the source, Dune will use that to fix the exact +version and source of dependencies. The default name of said lock directory is +`dune.lock`. Lock directories are created with: + +``` +$ dune pkg lock +Solution for dune.lock: +- ocaml.5.2.0 +- ocaml-base-compiler.5.2.0 +- ocaml-config.3 +``` + +Whenever Dune encounters a `dune.lock` folder, it will use the set of +dependencies defined in the lock. It contains all the metadata about package +names and versions, their dependencies and source locations that are necessary +to build the project's dependencies. + +On the next build, Dune will read the stored solver solution from the +`dune.lock` directory, download and build the dependencies and then continue on +building the project as usual. + +The lock directory will not be updated until `dune pkg lock` is rerun. + +:::{note} +This approach is similar to using `opam switch export --full --freeze` to +export the configuration of a switch. +::: + +Deleting the lock directory will cause Dune to fall back to automatically +determining dependency versions via the declared package constraints. diff --git a/doc/tutorials/dune-package-management/pinning.md b/doc/tutorials/dune-package-management/pinning.md index 58338c77fa7..4377fe435cb 100644 --- a/doc/tutorials/dune-package-management/pinning.md +++ b/doc/tutorials/dune-package-management/pinning.md @@ -26,28 +26,37 @@ dependencies. :::: -The next time the package is locked, Dune will use this repository instead of -the information from the selected package repositories. +The next time the project is built, Dune will use this repository instead of +the information from the selected package repositories. Thus the `fmt` package +will be built from the source in the specified Git repository rather than from +the source tarball released in the `opam-repository`. +```sh +dune build ``` -$ dune pkg lock -Solution for dune.lock: -- base-unix.base -- fmt.dev -- ocaml.5.0.0 -- ocaml-base-compiler.5.0.0 -- ocaml-config.3 -- ocamlbuild.0.15.0+dune -- ocamlfind.1.9.6+dune -- topkg.1.0.7 + +will find a new solution, quoting `_build/log`: + +``` +... + +# Dependency solution for +# _build/.sandbox//_private/default/.lock/dune.lock: +# - base-unix.base +# - fmt.dev +# - ocaml.5.4.0 +# - ocaml-base-compiler.5.4.0 +# - ocaml-compiler.5.4.0 +# - ocaml-config.3 +# - ocamlbuild.0.16.1+dune +# - ocamlfind.1.9.8+dune +# - topkg.1.1.1 +... ``` Unlike previously, the version of the `fmt` library that is picked is `dev`, to signify a development version. -The next time the project is built, the `fmt` package will be built from the -source in the specified Git repository rather than from the source tarball -released in the `opam-repository`. ``` $ dune exec ./test.exe diff --git a/doc/tutorials/dune-package-management/repos.md b/doc/tutorials/dune-package-management/repos.md index 94be14b653f..7070b4f44c9 100644 --- a/doc/tutorials/dune-package-management/repos.md +++ b/doc/tutorials/dune-package-management/repos.md @@ -17,40 +17,56 @@ if it didn't exist): :::{literalinclude} repos/dune-workspace :language: dune +:emphasize-lines: 5-6,8-10 ::: :::: In this case, we want to select a specific revision of the community repository instead of always using the most recent one as it would do by default. We -define a new repository and configure the lock directory to use this +define a new repository and configure the Dune solver to use this repository. For more information about the stanzas refer to the {doc}`repositories stanza ` as well as the {doc}`lock_dir stanza `. -When relocking the dependencies, the list of packages that are found as -dependencies changes accordingly: +The next time the build system is run, instead of using the default +repositories at their newest versions, the solver will check out the configured +repositories at the defined revisions. These will then be used to determine the +new solution, which will get used for downloading and building the +dependencies. +```sh +dune build ``` -$ dune pkg lock -Solution for dune.lock: -- base-unix.base -- fmt.0.9.0 -- ocaml.5.0.0 -- ocaml-base-compiler.5.0.0 -- ocaml-config.3 -- ocamlbuild.0.15.0+dune -- ocamlfind.1.9.6+dune -- topkg.1.0. -``` -Compared to before, the OCaml compiler version is older, which shows -that we did indeed pick an older version of the package repository for locking. +will thus log a new solution in `_build/log`. Note that a lot of package +versions are different, as the state of the opam-repository is frozen at the +specific commit: + +``` +... +# Dependency solution for +# _build/.sandbox//_private/default/.lock/dune.lock: +# - base-unix.base +# - fmt.0.9.0 +# - ocaml.5.0.0 +# - ocaml-base-compiler.5.0.0 +# - ocaml-config.3 +# - ocamlbuild.0.16.1+dune +# - ocamlfind.1.9.8+dune +# - topkg.1.0.7 +... +``` :::{note} This feature can also be used to make sure the locked dependencies are reproducible, as fixing all the package repository versions will lead to equivalent locking results. ::: + +:::{seealso} +{doc}`/tutorials/dune-package-management/locking` + More information how to keep package versions locked. +::: diff --git a/doc/tutorials/dune-package-management/repos/dune-workspace b/doc/tutorials/dune-package-management/repos/dune-workspace index bfe2c52908f..5e2f0e85664 100644 --- a/doc/tutorials/dune-package-management/repos/dune-workspace +++ b/doc/tutorials/dune-package-management/repos/dune-workspace @@ -1,4 +1,6 @@ -(lang dune 3.17) +(lang dune 3.21) + +(pkg enabled) (lock_dir (repositories overlay specific-upstream)) diff --git a/doc/tutorials/dune-package-management/setup.md b/doc/tutorials/dune-package-management/setup.md index 8430b176dbb..c98da062981 100644 --- a/doc/tutorials/dune-package-management/setup.md +++ b/doc/tutorials/dune-package-management/setup.md @@ -55,30 +55,26 @@ just define the module as an executable. After our project skeleton is set up, we can proceed to the next step. -## Locking Dependencies +## Enable package management -After declaring the dependencies, you will need to tell Dune which package -versions to use for your project. This is done by creating a lock directory. -This is easily done with a new Dune command: +So far we have done everything as in a regular Dune project. However, now we +need to tell Dune that we want to use the package management feature. To do so +we have to add a new stanza to our `dune-workspace` file, creating it if it +doesn't exist. -``` -$ dune pkg lock -Solution for dune.lock: -- ocaml.5.2.0 -- ocaml-base-compiler.5.2.0 -- ocaml-config.3 -``` - -This will update all the required opam repositories, use the newest version of -each and try to find a set of packages and versions that satisfy the -constraints that your project dependencies declare. +::::{dropdown} `dune-workspace` +:icon: file-code -:::{note} -The versions that get locked might be different from this tutorial, as we only -specified the lower bound of `ocaml`; barring any additional configuration, Dune -will pick the newest possible version for each dependency. +:::{literalinclude} setup/dune-workspace +:language: dune +:emphasize-lines: 2 ::: +This new stanza will tell Dune to determine the packages that your project +depends on, find the right versions, download, and build them on the next build. + +:::: + ## Build Project To build the project, you can just use the regular Dune commands. @@ -87,11 +83,22 @@ To build the project, you can just use the regular Dune commands. dune build ``` -This will download, build, and install all your locked dependencies and then use -those to build your project. This means that the first time building it will take -longer than usual, as the dependencies need to be built first. Subsequent -builds where all dependencies have been built before will be just as fast as -before. +Since this is the first time we have run the build system after enabling package +management a number of things will happen: + + 1. It will download and update all required opam repositories to determine + which packages are available. + 2. It will attempt to find a package solution that satisfies all dependency + constraints. + 3. It will download the sources of the dependencies. + 4. It will build the dependencies in sandbox locations. + 5. It will install the dependencies in the build folder. + 6. It will build the project using the dependencies that it has installed. + +This means that building the first time will take longer than usual, as the +dependencies need to be built, possibly including the OCaml compiler. +Subsequent builds where all dependencies have already been built will +be significantly faster. We can show that the package has been built successfully and works as expected: @@ -101,13 +108,16 @@ Hello, OCaml, Rust! ``` :::{note} -If you want to only build and fetch the project dependencies, you can use +If you don't want to build your project, instead stopping at step 5, you can use the `@pkg-install` alias like so ```shell $ dune build @pkg-install ``` +This functionality can be useful to cache the installation of dependencies, +somewhat similar to `opam switch create` followed by `opam install`. + See {doc}`/reference/aliases/pkg-install` for more information. ::: diff --git a/doc/tutorials/dune-package-management/setup/dune-workspace b/doc/tutorials/dune-package-management/setup/dune-workspace new file mode 100644 index 00000000000..3562360fe0d --- /dev/null +++ b/doc/tutorials/dune-package-management/setup/dune-workspace @@ -0,0 +1,2 @@ +(lang dune 3.21) +(pkg enabled)