Skip to content

gitian-building.md

Latest commit

 

History

History
367 lines (271 loc) · 10.9 KB

gitian-building.md

File metadata and controls

367 lines (271 loc) · 10.9 KB

Gitian Building

Last updated: October 19, 2021

This is based on fanquake's gitian-building, a resource I began updating in January 2020 with info and changes that were helpful when using it for the first time. Since then, this is updated regularly.

If you have already set up your environment for gitian builds, you can skip directly to Make Base VM for Gitian Building below.

Bitcoin Core releases are reproducibly built using Gitian Builder.

Gitian is the deterministic build process that is used to build the Bitcoin Core executables. It provides a way to be reasonably sure that the executables are really built from the git source. It also makes sure that the same, tested dependencies are used and statically built into the executable.

Multiple developers build the source code by following a specific descriptor ("recipe"), cryptographically sign the result, and upload the resulting signature. These results are compared and only if they match, the build is accepted and provided for download.

More independent Gitian builders are needed, which is why this guide exists. It is preferred you follow the steps yourself instead of using someone else's VM image to avoid "contaminating" the build.

For further information, the Bitcoin Core "Gitian building" documentation is at https://github.com/bitcoin-core/docs/blob/master/gitian-building.md

Initial Dependencies

This is a simplified version of the dependencies documentation in https://github.com/devrandom/gitian-builder#prerequisites -- don't hesitate to refer to it for further info or if you get stuck.

Linux

If coreutils or ruby is already included in your distribution, remove it from this line:

sudo apt install apt-cacher-ng coreutils ruby

macOS

brew install coreutils ruby

Gitian-builder needs sha256sum, which doesn't exist on macOS.

sudo ln -s /usr/local/bin/gsha256sum /usr/local/bin/sha256sum
brew cask install docker && brew link docker

Docker info for all platforms

  • To install Docker, see the information for your platform here: Docker Engine overview.
  • For the build script below to function, Linux users may need to allow Docker to run as a non-root user.
  • Once Docker is installed, run it at least once (e.g. with docker run hello-world or by opening the app) so Docker can request and set up the necessary permissions.
  • You might want to give Docker access to more CPUs, RAM, etc. This can be done by going to Preferences -> Advanced in the app and allocating as required.

Initial Setup

mkdir gitian-building
pushd gitian-building

Fork and clone a copy of the gitian.sigs repository.

# After forking https://github.com/bitcoin-core/gitian.sigs on GitHub
git clone git@github.com:your_github_username/gitian.sigs.git

Clone the other required repositories.

git clone https://github.com/bitcoin/bitcoin.git
git clone https://github.com/devrandom/gitian-builder.git
git clone https://github.com/bitcoin-core/bitcoin-detached-sigs.git # sigs for macOS and Windows only

Download the gitian inputs/dependencies into gitian-builder.

pushd gitian-builder
make -C ../bitcoin/depends download SOURCES_PATH=`pwd`/cache/common
mkdir -p inputs
wget -P inputs https://bitcoincore.org/cfields/osslsigncode-Backports-to-1.7.1.patch
wget -O inputs/osslsigncode-2.0.tar.gz https://github.com/mtrojnar/osslsigncode/archive/2.0.tar.gz
wget -P inputs https://github.com/bitcoin/bitcoin/files/6175295/osslsigncode-1.7.1.tar.gz
wget -P inputs https://bitcoincore.org/depends-sources/sdks/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers.tar.gz
wget iP inputs https://bitcoincore.org/depends-sources/sdks/Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers.tar.gz

If building gitian for Bitcoin Core 0.20.x on macOS, you'll need:

wget -P inputs https://bitcoincore.org/depends-sources/sdks/MacOSX10.14.sdk.tar.gz

If building gitian for Bitcoin Core 0.19.x on macOS, you'll need:

wget -P inputs https://bitcoincore.org/depends-sources/sdks/MacOSX10.11.sdk.tar.gz

Exit when you're done with the downloads.

popd

Make Base VM for Gitian Building

pushd gitian-builder
git pull # pull any updates
bin/make-base-vm --suite bionic --arch amd64 --docker
popd

If you encounter an error about connection timeouts and are running a firewall like UFW, you may need to open port 3142, which is required for the apt-cacher-ng web server:

ufw allow 3142/tcp
ufw reload

Set up and check out the branches to build

You can run the following git tag command from within your local bitcoin repository to see the Bitcoin Core release tags and descriptions, ordered by most recent last.

pushd bitcoin
git tag -n | sort -V
popd

Update the version and signer values with the version to build and your username.

export VERSION=0.20.2
export SIGNER=your_username
export USE_DOCKER=1

pushd bitcoin
git checkout v$VERSION
popd

Build Unsigned Sigs

Since we'll be creating a PR to the gitian.sigs repo, make a new branch off master:

pushd gitian.sigs
git checkout -b $SIGNER-$VERSION-unsigned master
popd

All of the instructions in this section are to be run from within the gitian-builder directory:

pushd gitian-builder

Intro

The first time depends is built for a new version, it can take a long time, as dependencies are being built for all architectures and operating systems. Subsequent builds will be much faster, as only Bitcoin Core is being compiled.

If you encounter any errors, it may be helpful to restart the base VM, or more rarely, delete the cache in gitian-builder/cache and rebuild the depends:

# Don't do these unless you need to, because it takes much more time.
rm -r cache
make -C ../bitcoin/depends download SOURCES_PATH=`pwd`/cache/common

Follow build progress from the gitian-builder directory root using:

tail -f var/install.log # Setup
tail -f var/build.log # Building dependencies and Bitcoin Core

Examples

tail -f ~/projects/bitcoin/gitian-builder/var/install.log
tail -f ~/projects/bitcoin/gitian-builder/var/build.log

or

tail -f ~/projects/bitcoin/gitian-builder/var/*.log

Now, on to building the unsigned sigs

You would normally run all three of these (Linux, Windows, and macOS) and submit the results in one PR.

Adjust num-make (number of cores, like for make -j) and memory (RAM) as needed below. For instance, I have four cores and add one for the num_make value.

The first build will be slow. The subsequent ones will go faster.

Linux

bin/gbuild --num-make 5 --memory 12000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
bin/gsign --signer "$SIGNER" --release ${VERSION}-linux --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
mv build/out/bitcoin-*.tar.gz build/out/src/bitcoin-*.tar.gz ../

Windows

bin/gbuild --num-make 5 --memory 12000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
bin/gsign --signer "$SIGNER" --release ${VERSION}-win-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
mv build/out/bitcoin-*-win-unsigned.tar.gz inputs/bitcoin-win-unsigned.tar.gz
mv build/out/bitcoin-*.zip build/out/bitcoin-*.exe ../

macOS

bin/gbuild --num-make 5 --memory 12000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
mv build/out/bitcoin-*-osx-unsigned.tar.gz inputs/bitcoin-osx-unsigned.tar.gz
mv build/out/bitcoin-*.tar.gz build/out/bitcoin-*.dmg ../
popd

Commit Unsigned Sigs

pushd gitian.sigs
git add .
git commit -m "$SIGNER $VERSION unsigned"
git push -u origin $SIGNER-$VERSION-unsigned
popd

Go to https://github.com/bitcoin-core/gitian.sigs and open a PR from that commit. Done 🍻

Build Signed Sigs

Signed signatures can be built once the detached sigs are available in the detached-sigs repo. They are often pushed a day or so after the release is tagged and announced on the #bitcoin-core-dev IRC channel with a message like "22.0.0 detached signatures are up and tagged."

pushd bitcoin-detached-sigs
git pull # pull new detached signatures
popd

You would normally run both of these (macOS and Windows) and submit the results in one PR. These build far faster than the builds for the unsigned signatures -- within a minute or so, or even a few seconds.

Since we'll be creating a PR to the gitian.sigs repo, make a new branch off master:

pushd gitian.sigs
git checkout -b $SIGNER-$VERSION-signed master
popd

Then navigate to gitian-builder

pushd gitian-builder

macOS

bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
mv build/out/bitcoin-osx-signed.dmg ../bitcoin-${VERSION}-osx.dmg

Windows

bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
bin/gsign --signer "$SIGNER" --release ${VERSION}-win-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-signed ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
mv build/out/bitcoin-*win64-setup.exe ../bitcoin-${VERSION}-win64-setup.exe
popd

Commit Signed Sigs

pushd gitian.sigs
git add .
git commit -m "$SIGNER $VERSION signed"
git push -u origin $SIGNER-$VERSION-signed
popd

Go to https://github.com/bitcoin-core/gitian.sigs and open a PR from that commit. Done 🍻

One-off builds

You can invoke one-off builds using:

pushd gitian-builder

env USE_DOCKER=1 ./bin/gbuild -j8 -m 6000 \
--commit bitcoin=79218eae27f62e246d52dc6cda4e8846293e1c8e \
--url bitcoin=https://github.com/fanquake/bitcoin.git \
path/to/bitcoin/contrib/gitian-descriptors/gitian-osx.yml

Useful for testing PRs like #17787.