Skip to content

Raspberry Pi Setup

Jump to bottom
Jim Borden edited this page Aug 26, 2021 · 6 revisions

As we move forward with support for Raspberry Pi it will be important to get a clear process in place so that we can reliably know how to generate a good build environment for it. This guide will show how to build a cross-compilation environment. This is specific to Linux so you will either need to be running directly on Linux, using a docker image of Linux, or using WSL on Windows.

Part 1: Toolchain

Any compiler that emits binaries compatible with the arm-linux-gnueabi triple will do the job. aarch64-linux-gnu for the newer (3+) Raspberry Pi models. There is a repo which hosts premade GNU toolchains specifically for Raspberry Pi if you are running on Linux or WSL.

If running a Debian-derived build you can also, for example, run apt-get install g++-arm-linux-gnueabihf but you will miss out on some device specific flags if you want them (e.g. -mcpu, etc).

Extract those somewhere in your system path, or modify the system path to be able to find arm-linux-gnueabihf-gcc.

Part 2: Sysroot

NOTE: If you already have a sysroot generated you can skip this part. If using docker the container needs to be privileged and if using WSL the install needs to be WSL 2. Alternatively, you can install the OS onto an actual device and skip to Part 2b.

(What is a sysroot? To put it in easy terms: It means that you will set a directory and tell the compiler "this is the only place you are allowed to look for headers and libraries." This will effectively trick the compiler into thinking it is running in an actual Raspberry Pi environment. The sysroot contains subdirectories like usr/include and lib/arm-linux-gnueabihf that correspond to the ones found at the root of an actual Linux filesystem.)

To generate a sysroot, the high level steps are to

  1. obtain a disk image of Raspbian,
  2. modify it as necessary, and
  3. copy out the necessary libraries and C/C++ header files.

Part 2a: Modifying the .img file directly

This is slightly more complicated, but faster and easier to automate than using an actual device. The actual device steps will be in Part 2b.

Get Raspbian

The current way we do this is to make a build specific to a version of the OS on Raspberry Pi. Let's use Raspbian 9 as an example.

First, let's see what is available. If you need to use server / graphical libraries you will need the full version of Raspbian, but for our purposes we will use the Lite version which is only 1/3 of the size. A list of such images is available from the Raspberry Pi downloads server. I'll use the newest one which currently is raspbian_lite-2020-02-14. You can download via HTTP or via torrent.

After downloading, what you will have is a filed called 2020-02-13-raspbian-buster-lite.zip. Unzip this file somewhere. Now what you will have is a disk image containing the filesystem of Raspbian called 2020-02-13-raspbian-buster-lite.img. This needs to be mounted read-write.

# Install needed tools (e.g. on Debian / Ubuntu)
apt-get install -y kpartx parted e2fsck-static \
    qemu qemu-user-static binfmt-support xz-utils rsync

# extend raspbian image by 128MB
dd if=/dev/zero bs=1M count=128 >> 2020-02-13-raspbian-buster-lite.img

# set up image as loop device
# You will see output as in the following:
# add map loop0p1 (253:4): 0 85623 linear 7:4 8192
# add map loop0p2 (253:5): 0 11588248 linear 7:4 94208
# Remember which "loop" gets assigned, in this case loop0
kpartx -v -a 2020-02-13-raspbian-buster-lite.img

# Use the same loop from before
parted /dev/loop0
    resizepart 2 -1s # Expand partition 2 to fill the space extended above
    quit

# Recreate the loop device, be sure to run both of these
# so the device actually gets removed (Docker is funny?)
# confirm the output still shows loop0
kpartx -d /dev/loop0
losetup -d /dev/loop0
kpartx -v -a 2020-02-13-raspbian-buster-lite.img

# check file system
e2fsck -f /dev/mapper/loop0p2

# expand partition filesystem
resize2fs /dev/mapper/loop0p2

# mount at last
mkdir /mnt/raspbian
mount -o rw /dev/mapper/loop0p2  /mnt/raspbian
mount -o rw /dev/mapper/loop0p1 /mnt/raspbian/boot 

# optional, but recommended
mount --bind /dev/pts /mnt/raspbian/dev/pts

Modify Raspbian

Now what you have is the Raspberry Pi filesystem mounted into the directory /mnt/raspbian. This is where we are about to change our root to, and so there is one problem standing in the way. The ld.so.preload file on the filesystem contains libraries with absolute paths from the current root. However, none of them are needed for our purposes so we can just comment out every line:

# In the Docker container:
sed -i 's/^/#/g' /mnt/raspbian/etc/ld.so.preload

Because we installed qemu-user-static earlier, our system now has the ability to run executables of another architecture inline so long as they are statically linked, and only take dynamic links against certain system libraries. Combined with chroot this has the effect of turning the docker container into a makeshift Raspberry Pi emulator. If you want to confirm this for yourself, run this before the next step:

$ uname -m
x86_64

Now it's time for the magic command! Run chroot to change the system root to the Raspberry Pi filesystem:

$ chroot /mnt/raspbian/ /bin/bash

# Now try running uname again
$ uname -m
armv7l

Now we are emulating a Raspberry Pi environment, and so you can change it as you see fit and the image file that you copied way at the beginning will be updated to match. (However, we won't use the image file itself anymore.)

Part 2b: Using the actual device

If you install the .img onto an actual device instead, you can skip all of 2a and run the following on the device, otherwise continue to run it in your chroot from 2a.

Install Dependencies

For building, we need to install a few dependencies, so let's do that:

apt-get update
apt-get install -y libicu-dev zlib1g-dev

(Notice the "armhf" entries being installed, as apt-get automatically pulls the correct architecture into the correct place on the filesystem.)

Once that is done, simply exit out of the root (or exit out of your pi device), back to x64 land:

exit

Extract the libraries and headers

It's time to extract the sysroot from the filesystem. To do this we will use rsync on a few key directories, and make a compressed tar of the result. If you went the device route you will need to run one command on your device before proceeding so that rsync has proper access to copy the files:

# Allow any user to run rsync to copy files off of the device
echo "$USER ALL=NOPASSWD:$(which rsync)" | sudo tee --append /etc/sudoers

The following

mkdir /raspbian/sysroot
cd /raspbian/sysroot

# chroot version:
# The funny "dot" syntax above only copies from /usr and /lib
# as the "root" so the current folder will have those two
# folders
rsync -rzlR --progress \
    /mnt/raspbian/./usr/lib/arm-linux-gnueabihf/ \
    /mnt/raspbian/./usr/lib/gcc/arm-linux-gnueabihf/ \
    /mnt/raspbian/./usr/include/ \
    /mnt/raspbian/./lib/arm-linux-gnueabihf/
    .

# device version:
rsync -rzlR \
--progress --rsync-path="sudo rsync" \
pi@<ip>:/usr/lib/arm-linux-gnueabihf/ \
pi@<ip>:/usr/lib/gcc/arm-linux-gnueabihf/ \
pi@<ip>:/usr/include/ \
pi@<ip>:/lib/arm-linux-gnueabihf/ \
.

This will take a few minutes, and there is one more step to do afterwords. Linux seems to be a fan of making absolute path symlinks to some of its libraries. This is fine for the device, but it breaks the sysroot. These need to be changed to relative paths. Once again, the repo hosting the raspberry pi compilers comes to the rescue with a nice [python script](wget https://raw.githubusercontent.com/abhiTronix/rpi_rootfs/master/scripts/sysroot-relativelinks.py) that does this for us. Run it passing in the path to your sysroot as an argument.

Now you can tarball it, and skip this step in the future.

tar cfJv sysroot.tar.xz *

Cleanup (not needed for device)

Finally we clean up the Docker image by reverting the ld.so.preload fix and unmounting the Raspbian disk image. (Note that the loop devices persist after the Docker image is destroyed, so this is important,)

# In the Docker shell:
sed -i 's/^#//g' /mnt/raspbian/etc/ld.so.preload
umount /mnt/raspbian/{dev/pts,dev,sys,proc,boot,}
kpartx -d /dev/loop0
losetup -d /dev/loop0

Part 3: Cross Compiling

There is some setup that needs to be done, but fortunately it can all be relegated to CMake toolchains files. Since CMake is the official way to build Couchbase Lite for Raspberry Pi, just use this toolchain file (be sure to pick up the cross-armhf and cross-base files as well, since this includes them) in your CMake invocation. For example, if you make a project called hello cmake will create an executable called hello, which you can verify is an ELF executable:

$ file hello                                                         
hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV) <...>

And you will be able to successfully execute it on a Raspberry Pi device

Clone this wiki locally