Build reproducibility for automated release signing

[mengw15]

Background

While moving our release signing to a policy-compliant setup (see PJ's mail on
dev@, "signing key used for create-release-candidate.yml"), one precondition of
the ASF automated-signing policy is that all signed artifacts can be built
reproducibly — bit-for-bit — so anyone can independently rebuild them and
byte-compare against what CI produced. That is what makes a machine's signature
auditable: if what you built and what CI built are identical, the signature
attests to verifiable bytes.
(Policy: https://infra.apache.org/release-signing.html#automated-release-signing)

Where Texera stands

Good news first: both voted artifacts (the source tarball and the
docker-compose bundle) are repackagings of committed files — nothing is
compiled and no dependencies are resolved at build time — so this is far
easier for us than for projects shipping binaries. (Transitive dependencies
only come into play for the docker images, which are convenience binaries
outside the signed set.)

Today the artifacts are not reproducible, purely for packaging-metadata
reasons:

• plain tar -czf picks up filesystem file ordering and the runner's uid/gid,
• gzip embeds a compression timestamp,
• the compose bundle's processed files (e.g. the generated .env) carry fresh
  mtimes.

Demonstration — same commit (release/v1.3 tip), the current packaging run
twice:

1d07e1728727…   first run
7706218a6fd3…   second run   <- different bytes

versus a one-step git archive --format=tar.gz, run twice:

4c7851fe0e03…   first run
4c7851fe0e03…   second run   <- identical

Proposal

This looks fixable with a small, contained change:

1. Source tarball: produce it in one step with git archive (deterministic by
   construction — git tree order, commit-time mtimes, no gzip timestamp).
2. Compose bundle: deterministic GNU tar flags
   (--sort=name --owner=0 --group=0 --mtime=@<commit-ts>) plus gzip -n.
3. A verify-reproducible script so anyone can rebuild both artifacts from a
   tag and byte-compare against the staged ones — this becomes the policy's
   "validate on trusted hardware" step before promotion.
4. Optionally a CI job that builds everything twice and fails on any byte
   difference, so reproducibility stays enforced.

Scope note: this covers the voted artifacts. Docker images are convenience
binaries outside the signed set; making them reproducible is a much bigger
topic and not required by the policy for our case.

Happy to put up the PR if this sounds right — feedback welcome.

[parshimers]

I think this follows even for the docker images, or maybe especially for them. So, it isn't just constrained to the source tarball and compose files. Rather, it should be that you can produce absolutely everything from that source tarball.
https://reproducible-builds.org/docs/jvm/ has some good reference regarding this. It seems sbt has some tooling available already.
Another important thing, is that for release branches (and honestly I think for main as well), everything external that is depended on needs to be at a version. It can't be latest, a nightly, or a snapshot.
However I'm less sure about NPM and the JS ecosystem in general. If someone with more expertise in that area could chime in about that, it would be appreciated.

[mengw15]

Thanks Ian, good point on the images.

Two questions:

1. For the signing precondition, is it enough that the artifacts we actually sign and vote on — the source tarball and the compose bundle — are bit-reproducible? They reference images but don't contain them, and the images are convenience binaries outside the signed set.

2. Who decides that for us? Is it something you or the IPMC would weigh in on, or does it get settled when we write to security@ for the CI signing key?

[parshimers]

I suppose it all lies in how you interpret this, from the automated signing documentation:

The release process contains a validation step where all artifacts are reproduced and validated on trusted hardware (i.e. not GitHub Actions) before publication to pages intended for end users. This means rebuilding the artifacts from source and validating the resulting artifacts are bit-by-bit identical to those staged for release.

The way I interpret that is: Whatever is released, signed or not, has to be bit-for-bit reproducible if you want to release it through hardware that isn't owned by you, personally, as a committer. Otherwise there's no way to tell if something has been done to the artifact to compromise it on this hardware that you don't have full control over.

[parshimers]

I opened a draft PR for this, it's just a take at getting Claude to do it: #8812. To be completely clear: I'm not doing this because it's somehow a Mentor's job to do it, or that you might expect a Mentor to handle things like this. Quite to the contrary, it is the incubating project's burden to deal with making releases compliant with ASF policies. I'm just doing it (as a contributor) because I think it's an interesting problem that I hadn't fully grasped before, and one I will also have to solve in AsterixDB. So, Texera can be the guinea pig ;) I would also appreciate any help with it, if someone likewise finds the topic interesting.

Some of the things I don't quite understand yet, like how purportedly an image which installs something using a package manager is indeed deterministic. I can see how that could be true, but I can also see how it wouldn't be true.

[parshimers]

OK, I looked a bit more at #8812 and I understand it somewhat now.
The very clever solution Claude pulled out here is to use the ubuntu/debian snapshot archives, and date gating on PyPI, to basically guarantee a set of packages that go into one of our dockerfiles can be deterministic. I was thinking version pinning would be the way to go about this but apparently the packages can change sometimes even if the versions are the same.
I think it needs some more manual testing though. I also still don't entirely get why none of the things I would have thought would be necessary for the Scala part of the build seem to be. I guess somehow we don't run into some of the usual things that make jars non-deterministic from source, but I still would like to fully understand how.

[parshimers]

OK, I think (at least one) problem was the fact that find outputs things in an arbitrary order, and that was being used in the process of building the WF coordinator container.
@mengw15 would you mind rebuilding (with the same epoch) and sharing the hashes here? Maybe just for the ones that didn't match.

[mengw15]

Sure, happy to. Just to make sure we build the same thing: commit 7f627c0b2, and by the same epoch do you mean last time's SOURCE_DATE_EPOCH=1791491650 with PACKAGE_SNAPSHOT=2026-10-08, or the new commit's time (1792092346, i.e. 2026-10-15)?

[parshimers]

The former (the epoch and PACKAGE_SNAPSHOT we were using this whole time). That way we can know better that it actually will work for some date relatively in the past, and not just in the past day or so.

[mengw15]

Thanks! Rebuilt 7f627c0b2 with SOURCE_DATE_EPOCH=1791491650 and PACKAGE_SNAPSHOT=2026-10-08 (BuildKit v0.33.1, clean worktree, --no-cache, linux/amd64).

texera-workflow-execution-coordinator, identical across two back-to-back builds:

• config 300b138b765a (sha256:300b138b765a0fb5c453bb886e42a3e0ce428310615f376f745c2b614be87bcb)
• manifest sha256:6b237b00e4a688d1e972296330c2df3112d258915dc94da2e2b5f1628da8d4a1

y-websocket-server doesn't reproduce even against itself: two back-to-back builds on my machine differ only in the npm ci layer, and only in caches written during the install: 543 files under /tmp/node-compile-cache/ and 39 under /root/.npm/ (cache index entries record a write time, and the debug log is named after the clock). node_modules is identical. With RUN npm ci && rm -rf /root/.npm /tmp/node-compile-cache, two builds come out identical (config f83b6f7427fa, manifest sha256:3a853e79e14d19edbab3f9ea717d3aae945db00272096d245bf3ad84d3b9d4d8). Its FROM node:24.19.0-slim isn't pinned to a digest yet either.

[parshimers]

Great, I was able to reproduce the workflow-execution-coordinator now. So, getting closer.
The y-websocket-server must have been missed. We release this image too, right? Let me see about making it reproducible.

[parshimers]

OK, now with the latest commit (ccc5da1), I think all the containers and tarballs should be reproducible with the correct PACKAGE_SNAPSHOT and SOURCE_DATE_EPOCH set.
With the latest commit, and SOURCE_DATE_EPOCH=1791491650 & PACKAGE_SNAPSHOT=2026-10-08 , for y-websocket-server I get:
manifest 92f8588c619170dd718fa15184786828e8afb38f88fb08ff03f25d4e0760bb0f
config 17decee7f785a9fdf244930685c33f7313f14be08426ba40b246af4bf2ea2082

If that one passes, I think we can do one last round of review in PR #8812 and then merge it, since we can reasonably infer the rest of the containers that were reproducible will remain so.

[mengw15]

Matches here too: y-websocket-server at ccc5da1f3 with SOURCE_DATE_EPOCH=1791491650 and PACKAGE_SNAPSHOT=2026-10-08 gives manifest sha256:92f8588c619170dd718fa15184786828e8afb38f88fb08ff03f25d4e0760bb0f and config sha256:17decee7f785a9fdf244930685c33f7313f14be08426ba40b246af4bf2ea2082, identical across two --no-cache builds.