Skip to content

Kernel

Jump to bottom Edit New page
Fangrui Song edited this page May 27, 2025 · 4 revisions

Basics

Clone the mainline kernel: git clone https://github.com/torvalds/linux.git --single-branch (usually faster than https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git)

Install packages needed for building a kernel.

sudo apt install libncurses-dev flex bison openssl libssl-dev \
                 dkms libelf-dev libudev-dev libpci-dev       \
                 libiberty-dev autoconf

make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig
make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 bzImage modules
scripts/clang-tools/gen_compile_commands.py -d /tmp/linux/x86_64  # For ccls

To build the kernel with Clang, lld, and LLVM binary utilities, build the tools in a llvm-project build directory.

ninja -C /tmp/Rel clang lld llvm-{ar,nm,objcopy,objdump,ranlib,readelf,strings,strip}

PATH=/tmp/Rel/bin:$PATH make O=/tmp/linux/x86_64-llvm ARCH=x86_64 LLVM=1 -j60 defconfig all

Clone https://github.com/ClangBuiltLinux/boot-utils

~/Dev/ClangBuiltLinux/boot-utils/boot-qemu.py -a x86_64 -k /tmp/linux/x86_64-llvm

Building perf

PATH=/tmp/out/custom1:$PATH make O=/tmp/linux/perf ARCH=x86_64 -j60 -C tools/perf EXTRA_CFLAGS='-O0 -g' CC='clang -w' DEBUG=1

Note: don't use a kernel build directory.

Building uAPI header files

https://www.kernel.org/doc/Documentation/kbuild/headers_install.txt

make ARCH=i386 INSTALL_HDR_PATH="$pkgdir/usr" headers_install

Testing

Linux Kernel Functional Testing: https://lkft.linaro.org/

initramfs

Use dracut to build an initramfs.

Use lsinitramfs path/to/initramfs (from initramfs-tools) to list files in an initramfs.

Attaching GDB to QEMU

scripts/config --file /tmp/linux/x86_64/.config -d RANDOMIZE_BASE # or specify noaslr in the kernel command line
scripts/config --file /tmp/linux/x86_64/.config -d CPU_SRSO -d CPU_UNRET_ENTRY # srso_return_thunk & retbleed_return_thunk make debugging hard
scripts/config --file /tmp/linux/x86_64/.config -e DEBUG_INFO_DWARF5 -e DEBUG_INFO_SPLIT -e GDB_SCRIPTS -e KGDB

scripts/config --file /tmp/linux/x86_64/.config -e TRANSPARENT_HUGEPAGE -e TRANSPARENT_HUGEPAGE_MADVISE -e READ_ONLY_THP_FOR_FS

make O=/tmp/linux/x86_64 -j60 scripts_gdb

Set GDB add-auto-load-safe-path

cgdb -ex 'tar rem :1234' /tmp/linux/x86_64/vmlinux
(gdb) hb start_kernel
(gdb) c

Breakpoints do not work reliably. Consider adding __attribute__((optimize("O0"))) (GCC) or __attribute__((optnone)) (Clang) to interesting functions.

Installing a Linux distributions

aarch64 port

In the kernel source directory, run

make O=/tmp/linux/arm64 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j60 defconfig all

Then create an Alpine aarch64 image following https://hackmd.io/@starnight/Run_Alpine_on_QEMU_aarch64_Virtual_Machine

wget https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/aarch64/netboot/vmlinuz-lts https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/aarch64/netboot/config-lts

qemu-img create -f qcow2 alpine-aarch64.qcow2 16G
qemu-system-aarch64 -smp 2 -M virt -cpu cortex-a57 -m 1G -nographic \
  -kernel vmlinuz-lts -initrd initramfs-lts \
  --append "console=ttyAMA0 ip=dhcp alpine_repo=http://dl-cdn.alpinelinux.org/alpine/latest-stable/main/ modloop=http://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/aarch64/netboot/modloop-lts" \
  -hda alpine-aarch64.qcow2 \
  -netdev user,id=unet -device virtio-net-device,netdev=unet -net user

On the guest,

# user: root, no password
setup-alpine  # install to vda

On the host,

sudo modprobe nbd max_part=8
sudo qemu-nbd -c /dev/nbd0 alpine-aarch64.qcow2
sudo mount /dev/nbd0p1 /mnt/
cp /mnt/vmlinuz-lts ./vmlinuz-lts.img
sudo chmod o+r /mnt/initramfs-lts
cp /mnt/initramfs-lts ./initramfs-lts.img
sudo umount /dev/nbd0p1
sudo qemu-nbd -d /dev/nbd0
qemu-system-aarch64 -smp 2 -M virt -cpu cortex-a57 -m 1G -nographic \
  -kernel vmlinuz-lts.img -initrd initramfs-lts.img \
  --append "console=ttyAMA0 root=/dev/vda3 rw rootfstype=ext4" \
  -hda alpine-aarch64.qcow2 \
  -device e1000,netdev=net0 \
  -net nic -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=net0

To debug a just built kernel, replace -kernel vmlinuz-lts.img with -kernel /tmp/linux/arm64/arch/arm64/boot/Image and add -s -S. Then, in another terminal, run

gdb-multiarch /tmp/linux/arm64/vmlinux
(gdb) tar rem :1234
(gdb) hb start_kernel
(gdb) c

riscv port

sudo apt install qemu-system-misc

s390x port

wget https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/s390x/netboot/vmlinuz-lts https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/s390x/netboot/initramfs-lts
qemu-img create -f qcow2 alpine-s390x.qcow2 6G
qemu-system-s390x -M s390-ccw-virtio -m 1024 -smp 2 -nographic -hda alpine-s390x.qcow2 -kernel vmlinuz-lts -initrd initramfs-lts \
  -append "alpine_repo=https://dl-cdn.alpinelinux.org/alpine/latest-stable/main modloop=https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/s390x/netboot/modloop-lts"

Run setup-alpine. After installation, run:

sudo modprobe nbd max_part=8
sudo qemu-nbd -c /dev/nbd0 alpine-s390x.qcow2
sudo mount /dev/nbd0p1 /mnt/
cp /mnt/vmlinuz-lts ./vmlinuz-lts.img
sudo chmod o+r /mnt/initramfs-lts
cp /mnt/initramfs-lts ./initramfs-lts.img
sudo umount /dev/nbd0p1
sudo qemu-nbd -d /dev/nbd0
qemu-system-s390x -smp 2 -M s390-ccw-virtio -m 1G -nographic -kernel vmlinuz-lts.img -initrd initramfs-lts.img \
  --append "console=ttyS0 root=/dev/vda3 rootfstype=ext4 rw" \
  -drive file=alpine-s390x.qcow2,if=virtio \
  -fsdev local,id=t,path=/tmp/t,security_model=none -device virtio-9p-pci,fsdev=t,mount_tag=t

x86_64 port

Download a standard ISO from https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/x86_64/.

qemu-img create -f qcow2 alpine-x86_64.qcow2 8G
qemu-system-x86_64 -enable-kvm -smp 2 -cpu host -m 1G -nographic \
  -nic user -boot d -cdrom alpine-standard-3.19.0-x86_64.iso -hda alpine-x86_64.qcow2

Run setup-alpine. After installation, run:

sudo modprobe nbd max_part=8
sudo qemu-nbd -c /dev/nbd0 alpine-x86_64.qcow2
sudo mount /dev/nbd0p1 /mnt/
cp /mnt/vmlinuz-lts ./vmlinuz-lts.img
sudo chmod o+r /mnt/initramfs-lts
cp /mnt/initramfs-lts ./initramfs-lts.img
sudo umount /dev/nbd0p1
sudo qemu-nbd -d /dev/nbd0
qemu-system-x86_64 -enable-kvm -smp 2 -cpu host -m 1G -nographic -kernel vmlinuz-lts.img -initrd initramfs-lts.img \
  --append "console=ttyS0 root=/dev/vda3 rootfstype=ext4 rw" \
  -drive file=alpine-x86_64.qcow2,if=virtio \
  -device e1000,netdev=net0 -net nic -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=net0 \
  -fsdev local,id=t,path=/tmp/t,security_model=none -device virtio-9p-pci,fsdev=t,mount_tag=t

On the guest,

mkdir /tmp/t && mount -t 9p t /tmp/t -o trans=virtio,msize=$((5*1024*1024))

To debug a just built kernel, replace -kernel vmlinuz-lts.img with -kernel /tmp/linux/x86_64/arch/arm64/boot/Image and add -s -S. Then, in another terminal, run

gdb-multiarch /tmp/linux/arm64/vmlinux
(gdb) tar rem :1234
(gdb) hb start_kernel
(gdb) c

Old notes on Fedora guest

sudo sshfs -p 2222 -o allow_other root@localhost:/ /mnt/fedora   # Add `PermitRootLogin yes` to guest /etc/ssh/sshd_config
make -j$(nproc) O=/tmp/linux/x86_64 INSTALL_PATH=/mnt/fedora/boot install
make -j$(nproc) O=/tmp/linux/x86_64 INSTALL_MOD_PATH=/mnt/fedora modules_install

In the guest, create a new entry under /boot/loader/entries/ based on an existing one. Reuse initramfs but update vmlinuz.

vim /etc/default/grub  # Add nokaslr to GRUB_CMDLINE_LINUX
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo grubby --set-default /boot/vmlinuz-6.*  # Select the new vmlinuz

(grubby doc) (If the guest is Debian, do sudo update-initramfs -k all -c; sudo update-grub)

Alternatively, append -kernel /tmp/linux/x86_64/arch/x86/boot/bzImage -initrd path/to/initramfs -append 'root=/dev/mapper/fedora_ju1f2mb4--trk--14--1-root ro rd.lvm.lv=fedora_ju1f2mb4-trk-14-1/root rhgb quiet nokaslr' to the qemu command line. Append -nographic -append 'tty=consoleS0 ... to avoid video output (a SDL window). (The virtual serial port and the QEMU monitor are multiplexed onto stdio.)

qemu-system-x86_64 -enable-kvm -nographic -m 16384 -smp 8 -cpu host -kernel /tmp/linux/x86_64/arch/x86/boot/bzImage -initrd /tmp/c/initramfs-5.19.7-300.fc37.x86_64.img -append 'console=ttyS0 root=/dev/mapper/fedora_ju1f2mb4--trk--14--1-root ro rd.lvm.lv=fedora_ju1f2mb4-trk-14-1/root rhgb quiet nokaslr' -drive file=fedora.img,if=virtio -net nic,model=virtio -nic user,hostfwd=tcp::2222-:22

Fedora uses /dev/zram0 as a swap device by default (https://fedoraproject.org/wiki/Changes/SwapOnZRAM). See man zranctl for other distributions. Run cat /proc/swaps (or swapon --show) to list swap devices.

kbuild performance

https://lkml.org/lkml/2018/5/2/74 from Ingo Molnar:

First I make sure that cpufreq is set to 'performance':

for ((cpu=0; cpu<120; cpu++)); do
  G=/sys/devices/system/cpu/cpu$cpu/cpufreq/scaling_governor
  [ -f $G ] && echo performance > $G
done

Then I copy a kernel tree to /tmp (ramfs) as root:

cd /tmp
rm -rf linux
git clone ~/linux linux
cd linux
make defconfig >/dev/null

... and then we can build the kernel in such a loop (as root again):

perf stat --repeat 10 --null --pre		'\
  cp -a kernel ../kernel.copy.$(date +%s);	 \
  rm -rf *;					 \
  git checkout .;				 \
  echo 1 > /proc/sys/vm/drop_caches;		 \
  find ../kernel* -type f | xargs cat >/dev/null;  \
  make -j kernel >/dev/null;			 \
  make clean >/dev/null 2>&1;			 \
  sync						'\
  						 \
  make -j16 >/dev/null
Add a custom footer
Add a custom sidebar
Clone this wiki locally