Protoc Dependency + HostBuildDependency build failures

[jeremiahpslewis]

Every few builds, a target platform fails randomly, with the following message seeming to hint at the issue:

→ [c7845625] + protoc_jll v105.29.3+0
—
  | Info packages marked with → not downloaded, use `instantiate` to download
  | Updating `/cache/build/yggy-amdci7-6/julialang/yggdrasil/V/Valhalla/build/armv7l-linux-gnueabihf-cxx03/ji8laS1V/x86_64-linux-musl-cxx11/.project/Manifest.toml`
  | [692b3bcd] + JLLWrappers v1.7.0
  | [21216c6a] + Preferences v1.4.3
  | [458c3c95] + OpenSSL_jll v3.0.15+3
  | → [c7845625] + protoc_jll v105.29.3+0

What causes packages not to be downloaded in CI? This platform built fine (with a downloaded binary) the previous build...

[giordano]

Reference to the failed job? From what you show here there's no error.

[jeremiahpslewis]

Failed job here: https://buildkite.com/julialang/yggdrasil/builds/17199#0194d510-520b-477b-911e-cc976cd143fb
In the previous build (with no relevant changes for the platform) it passed: https://buildkite.com/julialang/yggdrasil/builds/17197#0194d38c-3e3e-4ccb-bcac-a108d1d1c782

[giordano]

I'm missing the evidence that the failure you see is related to the message you posted at the top (which is not an issue), care to elaborate?

[jeremiahpslewis]

The actual error seems to be

[07:45:47] Error loading shared library libabsl_malloc_internal.so.2401.0.0: Exec format error (needed by /workspace/x86_64-linux-musl-cxx11/destdir/bin/protoc)

But I have no idea why this error would show up intermittently (nothing in the code relevant to the platform changed, just added an if mingw)

[eschnett]

The error above ("exec format error") is due to a confusion between build and host system. If you need to run an executable at build time then you need to have a BuildDependency present, not a Dependency.

Maybe you have this package present as both build dependency and regular dependency, and it's random in which order they appear on your path?

[jeremiahpslewis]

Sounds plausible! In my case it would be protoc_jll. Does a fix / kludge to prevent this come to mind?

[giordano]

The PATH isn't random, it's deterministically set in https://github.com/JuliaPackaging/BinaryBuilderBase.jl/blob/9b5e88dacc55c3bdffb59c11864676d66dcf0599/src/Runner.jl

[jeremiahpslewis]

Hit this intermittent error again: https://buildkite.com/julialang/yggdrasil/builds/17560#0194f4fb-6cbf-470e-94e2-f3cad62c587c

[12:44:04] Error loading shared library libabsl_strings.so.2401.0.0: Exec format error (needed by /workspace/x86_64-linux-musl-cxx11/destdir/bin/protoc)

[giordano]

Did you try to debug and reproduce this locally? Also, I still haven't got any answer to #10434 (comment). I'm very much not convinced anything in #10434 (comment) is relevant to whatever issue you're observing, packages not being "installed" according to Pkg is totally normal and expected, nothing wrong with that.

[jeremiahpslewis]

I'm missing the evidence that the failure you see is related to the message you posted at the top (which is not an issue), care to elaborate?

I don't have any evidence, it just seems correlated.

[jeremiahpslewis]

I've never encountered this error locally. Just ran the x86_64-linux-musl-cxx11 platform locally twice, built both times without error.

[imciner2]

The error above ("exec format error") is due to a confusion between build and host system. If you need to run an executable at build time then you need to have a BuildDependency present, not a Dependency.

Um, not quite. BuildDependency is going to tell the resolver that this package is only needed in the build environment in Ygg and should not be included as a dependency in the generated JLL package.

What you actually should use if you need to run a program during build is HostBuildDependency. That will ensure the build always downloads the linux musl version of the dependency so it can run in the sandbox.

[jeremiahpslewis]

Hit this issue again, this time:

[14:58:36] Error loading shared library libabsl_bad_variant_access.so.2401.0.0: Exec format error (needed by /workspace/x86_64-linux-musl-cxx11/destdir/bin/prot
oc)

https://buildkite.com/julialang/yggdrasil/builds/17723#019504cf-340c-41e3-9c6c-01135b673ad8

[jeremiahpslewis]

I think the issue may have something to do with having a compat flag for protoc as a Dependency but no flag for protoc as a HostBuildDependency, rerunning now...

[jeremiahpslewis]

Update…doesn’t help