build-env: doesn't seem to be honoured

[dbuenzli]

I have an opam file with:

build-env: CAML_LD_LIBRARY_PATH += "_build/src/std"

My build fails at the install: step with:

[…]
# Fatal error: cannot load shared library dllb0_stubs
# Reason: dlopen(dllb0_stubs.so, 0x000A): tried: 'dllb0_stubs.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OSdllb0_stubs.so' (no such file), '/usr/lib/dllb0_stubs.so' (no such file, not in dyld cache), 'dllb0_stubs.so' (no such file)

If I check the suggested build environment file, the environment variable doesn't seem to be setup as expected by the build-env field:

> cat ~/.opam/log/b0-51512-5d52f8.env | grep CAML_LD_LIBRARY_PATH
CAML_LD_LIBRARY_PATH=/Users/dbuenzli/.opam/ocaml.5.3.0+bco/lib/stublibs:/Users/dbuenzli/.opam/ocaml.5.3.0+bco/lib/ocaml/stublibs:/Users/dbuenzli/.opam/ocaml.5.3.0+bco/lib/ocaml

(Don't know if this is relevant but this package install uses both an .install file and an install: section).

> opam --version
2.3.0

[kit-ty-kate]

I was able to reproduce. However weirdly enough, it only seems to happen when trying to modify a variable that is already modified by another opam package (such as CAML_LD_LIBRARY_PATH or OCAML_TOPLEVEL_PATH)

[dbuenzli]

I didn't fully investigate why but it seems that in the build: section I have no problem running bytecode executables with C stubs, the dll are found. This was not the case in install:, hence my attempts at tweaking CAML_LD_LIBRARY_PATH which are no longer needed (see first message).

[dbuenzli]

hence my attempts at tweaking CAML_LD_LIBRARY_PATH which are no longer needed (see first message).

That seems to have been a "it works on my machine" moment. In fact I do need this to run bytecode executable with C stubs in build: aswell. So it would be nice to have a fix.

[kit-ty-kate]

I did some more debug in an empty switch:

• if you install package b with setenv: CAML_LD_LIBRARY_PATH = "b" it'll be always b regardless of build-env
• if you install package b with setenv: CAML_LD_LIBRARY_PATH += "b" it'll be b, then if package a has build-env: CAML_LD_LIBRARY_PATH += "a" it'll be b:a
  • same with =+
• the opposite happens (result is a:b if b has setenv: CAML_LD_LIBRARY_PATH =+ "b")

So something seems clear:

• setenv is (re)applied after build-env
• = overwrites other variables (that is expected)

The first point is the source of the problem and the interaction between build-env and setenv is "unspecified" in the manual. The best definition we have is between different setenv and this one is also unspecified:

Note that while it is guaranteed that dependents will see the environment updates, and dependencies won't, other cases are unspecified. The order in which setenv: updates done by different packages are applied is deterministic, but also unspecified. In particular, it is not currently guaranteed to follow dependency order.

I think we should try to have a better specification for it. On the spot, here what i think the broad specification could be:

• build-env is always applied after setenv
• setenv order is the following:
  • for-each environment variable:
    • lookup the = (overrides) first, apply them (order them by dependency and then unspecified if they aren't related or if there is a cycle in the case of post dependencies)
    • lookup the +=/+=/:=/=: (same, order by dependencies and then unspecified)
    • =+=: no idea on this one, it's easy if the previous operation was an = and easy enough if it had only one +=/etc but i have no idea what to do if there have been more than one of these

[kit-ty-kate]

I have a tentative fix in #6713 but the code needs to be polished first. With the freeze date of opam 2.5 so close i can't guarantee it'll make the cut for the alpha but it might get in during the alpha period as i personally think this is a bug fix.