Dune should run itself when a package build command tries to run `dune`

[gridbugs]

If the dune executable currently running isn't what dune resolves to in PATH, packages that invoke dune in their build command should still start a copy of the current dune rather than the one from PATH. This situation arises when running a development version of dune where the exe is named main.exe.

[rgrinberg]

note that this should extend to (run sh -c dune) as well. In other words, we should set PATH so that the current dune is included.

[gridbugs]

Good point; it's not sufficient to detect when the first word of the command is dune and replace it with the value of argv[0]. Setting PATH is also insufficient as it wouldn't handle the case where your dune binary is named something else (e.g. if you build dune yourself and it's still in _build/default/bin/main.exe). What about creating a temporary directory somewhere and copying/linking the current dune exe there (named "dune") and then putting that directory at the start of PATH? Is that too unix-specific for it to work on windows?

[dra27]

@gridbugs - that trick is entirely fine for Windows and indeed opam’s prototype compiler packaging is using a very similar trick to shim GCC out of Cygwin

[Leonidas-from-XIV]

You can get the current executable also on Windows like this (from @dra27's relocatable compiler branch), however unlike Unix.readlink on /proc/self/exe probably not in pure OCaml.

[kit-ty-kate]

I'm not sure to understand what is the advantage of using dune recursively? How does this work if the constraint for dune doesn't match the current one? Isn't that also contrary to the idea of a lock file?

[gridbugs]

@kit-ty-kate good point. And the lockdir does contain a lockfile for dune when a project's dependencies depend on dune, so we should be building dune as part of the project's build and making the resulting exe available to all packages that depend on it. I wonder that's not currently happening in my experiments. I'll investigate further.

[rgrinberg]

What about creating a temporary directory somewhere and copying/linking the current dune exe there (named "dune") and then putting that directory at the start of PATH? Is that too unix-specific for it to work on windows?

This is likely the best.

so we should be building dune as part of the project's build and making the resulting exe available to all packages that depend on it

That can't work because the dune that will be build will read/write metadata in a different format.

[gridbugs]

That can't work because the dune that will be build will read/write metadata in a different format.

Which metadata files are you referring to? If it's just metadata files within the package it's building then what would be the problem?

[rgrinberg]

The dune-package fill produced when building a package must be compatible. This is what dune reads to make libraries, package sites, and other metadata available.

[gridbugs]

But why does the dune that's orchestrating the build need to read the dune-package files of individual packages from the project's dependencies? Dependencies don't need to build with dune at all so I don't see why dune can't treat the dune package as a foreign build system that happens to be called "dune".

[rgrinberg]

Which dune do you plan to use to build the packages that are inside the user's workspace? E.g.

$ dune build bin/foo.exe

Will that dune used to install dune, or the dune that the user already had?

[gridbugs]

For packages in the current workspace I think it makes sense to use the dune the user already has (ie. the one orchestrating the build). But for packages in the lockdir that build with dune, I'm considering treating their dune dependency the same way we could treat any other build dependency of a package, building the dune package of the appropriate version and making its binary available to build the package.

[rgrinberg]

For packages in the current workspace I think it makes sense to use the dune the user already has

Okay, so this dune will not be able to read the metadata generated by the inner dune. To discover the metadata about the packages we just built, we need to read the dune-package files.