ARM Assembly Cheat

Almost all the content in this repository has been moved to: https://github.com/cirosantilli/linux-kernel-module-cheat#userland-assembly

Only the core infrastructure of this repo was left here. No major new features are intended to be added here.

Notable advantages of LKMC repository include:

• a single unified cross arch setup for ARM and x86_64, with cross arch concepts all nicely factored out

• gem5 support. This is because we have integration of QEMU / gem5 / buildroot setups already done there

• parallel testing. Mostly because the build system there is Python, which is more flexible.

• other stuff I can’t remember right now. That setup just has a ton of features, and will continue to get more and more ;-)

There is only one use case left for this repository: since this is more minimal, it is easier to upgrdte to the latest binutils-gdb here without breaking unrelated stuff, in order to get very latest instructions.

The initial motivation for that was [sve] although that specific case has already been moved to LKMC as well!

I might however start taking some risks on LKMC and upgrading binutils-gdb to master when needed anyways, or just build the latest binutils-gdb myself over there for userland only.

1. Old README

Here is the old README with only infrastructure sections left.

ARMv7 and [ARMv8] assembly userland minimal examples tutorial. Runnable asserts on x86 hosts with QEMU user mode or natively on ARM targets. Nice GDB step debug setup. Tested on Ubuntu 18.04 host and Raspberry Pi 2 and 3 targets. Baremetal setup at: https://github.com/cirosantilli/linux-kernel-module-cheat#baremetal-setup x86 cheat at: https://github.com/cirosantilli/x86-assembly-cheat

• 1. Old README
• 2. Getting started
  • 2.1. Asserts
  • 2.2. Getting started on non-Ubuntu operating systems
    • 2.2.1. Other Linux distro hosts
      • 2.2.1.1. Docker host setup
    • 2.2.2. Non-Linux host
    • 2.2.3. Raspberry Pi 2 native
    • 2.2.4. Raspberry Pi 3 native
  • 2.3. GDB step debug
    • 2.3.1. Advanced usage
  • 2.4. Disassemble
  • 2.5. crosstool-NG toolchain
  • 2.6. Build the documentation
  • 2.7. Custom build flags
• 3. Theory
  • 3.1. Architecture of this repo
    • 3.1.1. C driver
• 4. CONTRIBUTING
  • 4.1. Update QEMU
  • 4.2. Bare metal

2. Getting started

On Ubuntu, clone, configure, build QEMU and Binutils from source, run all ARMv7 and ARMv8 examples through QEMU user, and assert that they exit with status 0:

git clone --recursive https://github.com/cirosantilli/arm-assembly-cheat
cd arm-assembly-cheat
./download-dependencies
make test
echo $?

Expected outcome: the exit status is successful:

0

For other operating systems, see: Getting started on non-Ubuntu operating systems.

We compile our own Binutils and QEMU to be able to use the newest ISA features. Those projects build pretty fast (~10 minutes), so it is fine. The cleanest thing would be to also compile GCC with crosstool-NG toolchain.

The armv7 examples are all located under the v7 directory. Run all of them:

cd v7
make test
echo $?

Run just one of them:

cd v7
make test-<basename-no-extension>
echo $?

E.g.:

make test-add

will run userland/arch/arm/add.S.

This just tests some assertions, but does not output anything. See: Asserts.

Alternatively, to help with tab completion, the following shortcuts all do the same thing as make test-add:

./t add
./t add.
./t add.out

[armv8] examples are all located under the v8 directory. They can be run in the same way as ARMv7 examples:

cd v8
make test-movk

Just build the examples without running:

make

Clean the examples:

make clean

This does not clean QEMU builds themselves. To do that run:

make qemu-clean

2.1. Asserts

Almost all example don’t output anything, they just assert that the computations are as expected and exit 0 is that was the case.

Failures however output clear error messages.

Try messing with the examples to see them fail, e.g. modify userland/arch/arm/add.S to contain:

mov r0, #1
add r1, r0, #2
ASSERT_EQ(r1, 4)

and then watch it fail:

cd v7
make test-add

with:

error 1 at line 12
Makefile:138: recipe for target 'test-add' failed
error 1 at line 12

since 1 + 2 tends to equal 3 and not 4.

So look how nice we are: we even gave you the line number 12 of the failing assert!

2.2. Getting started on non-Ubuntu operating systems

If you are not on an Ubuntu host machine, here are some ways in which you can use this repo.

2.2.1. Other Linux distro hosts

For other Linux distros, you can either:

• have a look at what download-dependencies does and adapt it to your distro. It should be easy, then proceed normally.

  Might fail due to some incompatibility, but likely won’t.

• run this repo with docker. Requires you to know some Docker boilerplate, but cannot (?) fail.

2.2.1.1. Docker host setup

sudo apt install docker
sudo docker create -it --name arm-assembly-cheat -w "/host/$(pwd)" -v "/:/host" ubuntu:18.04
sudo docker exec -it arm-assembly-cheat /bin/bash

Then inside Docker just add the --docker flag to ./download-dependencies and proceed otherwise normally:

./download-dependencies --docker
make test

The download-dependencies takes a while because build-dep binutils is large.

We share the repository between Docker and host, so you can just edit the files on host with your favorite text editor, and then just run them from inside Docker.

TODO: GDB TUI GUI is broken inside Docker due to terminal quirks. Forwarding the port and connecting from host will likely work, but I’m lazy to try it out now.

2.2.2. Non-Linux host

For non-Linux systems, the easiest thing to do is to use an Ubuntu virtual machine such as VirtualBox: https://askubuntu.com/questions/142549/how-to-install-ubuntu-on-virtualbox.

Porting is not however impossible because we use the C standard library for portability, see: Architecture of this repo. Pull requests are welcome.

2.2.3. Raspberry Pi 2 native

Yay! Let’s see if this actually works on real hardware, or if it is just an emulation pipe dream?

Tested on Raspbian Lite 2018-11-13 with this repo at commit bcddf29c8e00b30afe7b3643558b25f22a64405b.

For now, we will just compile natively, since I’m not in the mood for cross compilation hell today.

According to Wikipedia the Raspberry Pi 2 V 1.1 which I have has a BCM2836 SoC, which has 4 ARM Cortex-A7 cores, which implement ARMv7-A, VFPv4 and [neon].

Therefore we will only be able to run v7 examples on that board.

First connect to your Pi through SSH as explained at: https://stackoverflow.com/revisions/39086537/10

Then inside the Pi:

sudo apt-get update
sudo apt-get install git make gcc gdb
git clone https://github.com/cirosantilli/arm-assembly-cheat
cd arm-assembly-cheat/v7
make NATIVE=y test
make NATIVE=y gdb-add

GDB TUI is slightly buggier on the ancient 4.9 toolchain (current line gets different indentation, does not break on the right instruction after asm_main_after_prologue, cannot leave TUI), but it might still be usable

The Pi 0 and 1 however have a BCM2835 SoC, which has an ARM1176JZF-S core, which implements the ARMv6Z ISA, which we don’t support yet on this repo.

Bibliography: https://raspberrypi.stackexchange.com/questions/1732/writing-arm-assembly-code/87260#87260

2.2.4. Raspberry Pi 3 native

The Raspberry Pi 3 has a BCM2837 SoC, which has 4 Cortex A53 cores, which implement ARMv8-A.

However, as of July 2018, there is no official [armv8] image for the Pi 3, the same ARMv7 image is provided for both: https://raspberrypi.stackexchange.com/questions/43921/raspbian-moving-to-64-bit-mode

Then we look at the following threads:

• https://raspberrypi.stackexchange.com/questions/49466/raspberry-pi-3-and-64-bit-kernel-differences-between-armv7-and-armv8

• https://raspberrypi.stackexchange.com/questions/77693/enabling-armv8-on-raspberry-pi-3-b

which lead us to this 64-bit Debian based distro for the Pi: https://github.com/bamarni/pi64

So first we flash pi64’s 2017-07-31 release, and then do exactly the same as for the Raspberry Pi 2, except that you must go into the v8 directory instead of v7.

TODO: can we run the v7 folder in ARMv8? First I can’t even compile it. Related: https://stackoverflow.com/questions/21716800/does-gcc-arm-linux-gnueabi-build-for-a-64-bit-target For runtime: https://stackoverflow.com/questions/22460589/armv8-running-legacy-32-bit-applications-on-64-bit-os

2.3. GDB step debug

Debug one example with GDB:

make gdb-add

Shortcut:

./t -g add

This leaves us right at the end of the prologue of asm_main in GDB TUI mode, which is at the start of the assembly code in the .S file.

Stop on a different symbol instead:

make GDB_BREAK=main gdb-add

Shortcut:

./t -b main -g add

It is not possible to restart the running program from GDB as in gdbserver --multi unfortunately: https://stackoverflow.com/questions/51357124/how-to-restart-qemu-user-mode-programs-from-the-gdb-stub-as-in-gdbserver-multi

Quick GDB tips:

• print a register:

  i r r0

  Bibliography: https://stackoverflow.com/questions/5429137/how-to-print-register-values-in-gdb

• print floating point registers:

  • https://reverseengineering.stackexchange.com/questions/8992/floating-point-registers-on-arm/20623#20623

  • https://stackoverflow.com/questions/35518673/gdb-print-aarch64-advanced-simd-vector-registers-is-it-possible/54711214#54711214

• print an array of 4 32-bit integers in hex:

  p/x (unsigned[4])my_array_0

  Bibliography: https://stackoverflow.com/questions/32300718/printing-array-from-bss-in-gdb

• print the address of a variable:

  p &my_array_0

Bibliography: https://stackoverflow.com/questions/20590155/how-to-single-step-arm-assembler-in-gdb-on-qemu/51310791#51310791

2.3.1. Advanced usage

The default setup is opinionated and assumes that your are a newb: it ignores your .gdbinit and puts you in TUI mode.

However, you will sooner or later notice that TUI is crappy print on break Python scripts are the path of light, e.g. GDB dashboard.

In order to prevent our opinionated defaults get in the way of your perfect setup, use:

make GDB_EXPERT=y gdb-add

or the shortcut:

./t -G add

2.4. Disassemble

Even though GDB step debug can already disassemble instructions for us, it is sometimes useful to have the disassembly in a text file for further examination.

Disassemble all examples:

make -j `nproc` objdump

Disassemble one example:

make add.objdump

Examine one disassembly:

less -p asm_main add.objdump

This jumps directly to asm_main, which is what you likely want to see.

Disassembly is still useful even though we are writing assembly because the assembler can do some non-obvious magic that we want to understand.

2.5. crosstool-NG toolchain

Currently we build just Binutils from source, but use the host GCC to save time.

This could lead to incompatibilities, although we haven’t observed any so far.

crosstool-NG is a set of scripts that makes it easy to obtain a cross compiled GCC. Ideally we should track it here as a submodule and automate from there.

You can build the toolchain with crosstool-NG as explained at: https://stackoverflow.com/revisions/51310756/6

Then run this repo with:

make \
  CTNG=crosstool-ng/.build/ct_prefix \
  PREFIX=arm-cortex_a15-linux-gnueabihf \
  test \
;

2.6. Build the documentation

If you don’t like reading on GitHub, the HTML documentation can be generated from the README with:

make doc
xdg-open out/README.html

2.7. Custom build flags

E.g., to pass -static for an emulator that does not support dynamically linked executables like gem5:

make CCFLAGS_CLI=-static

3. Theory

3.1. Architecture of this repo

qemu-arm-static is used for emulation on x86 hosts. It translates ARM to x86, and forwards system calls to the host kernel.

OS portability is achieved with the C standard library which makes system calls for us: this would in theory work in operating systems other than Linux if you port the build system to them.

Using the standard library also allows us to use its convenient functionality such as printf formatting and memcpy to check memory.

Non-OS portable examples will be clearly labeled with their OS.

These examples show how our infrastructure works:

• fail.S

• userland/arch/arm/hello_driver.S

• hello_common.S

3.1.1. C driver

We link all examples against a C program: main.c. Sample simplified commands:

arm-linux-gnueabihf-gcc -c -o 'main.o' 'main.c'
arm-linux-gnueabihf-gcc -c -o 'sub.o' 'sub.S'
arm-linux-gnueabihf-gcc -o 'sub.out' 'sub.o' main.o

The C driver then just calls asm_main, which each .S example implements.

This allows us to easily use the C standard library portably: from the point of view of GCC, everything looks like a regular C program, which does the required glibc initialization before main().

4. CONTRIBUTING

4.1. Update QEMU

https://stackoverflow.com/questions/816370/how-do-you-force-a-makefile-to-rebuild-a-target

git -C qemu pull
make -B -C v7 qemu
make -B -C v8 qemu

If the build fails due to drastic QEMU changes, first do:

make qemu-clean

Then make sure that the tests still pass:

make test

4.2. Bare metal

This tutorial only covers userland concepts.

However, certain instructions can only be used in higher privilege levels from an operating system itself.

Here is a base setup ARM programming without an operating system, also known as "Bare Metal Programming": https://github.com/cirosantilli/linux-kernel-module-cheat/tree/7d6f8c3884a4b4170aa274b986caae55b1bebaaf#baremetal-setup

Features:

• clean crosstool-NG build for GCC

• C standard library powevered by Newlib

• works on both QEMU and gem5

Here are further links:

• generic:

  • https://stackoverflow.com/questions/38914019/how-to-make-bare-metal-arm-programs-and-run-them-on-qemu/50981397#50981397 generic QEMU question

  • https://github.com/freedomtan/aarch64-bare-metal-qemu/tree/2ae937a2b106b43bfca49eec49359b3e30eac1b1: -M virt UART bare metal hello world, nothing else, just works

  • https://github.com/bravegnu/gnu-eprog Not tested.

  • https://stackoverflow.com/questions/29837892/how-to-run-a-c-program-with-no-os-on-the-raspberry-pi/40063032#40063032 no QEMU restriction

  • https://github.com/cirosantilli/raspberry-pi-bare-metal-blinker minimal, but not very QEMU friendly however because hard to observe LED: https://raspberrypi.stackexchange.com/questions/56373/is-it-possible-to-get-the-state-of-the-leds-and-gpios-in-a-qemu-emulation-like-t

• raspberry PI:

  • https://raspberrypi.stackexchange.com/questions/34733/how-to-do-qemu-emulation-for-bare-metal-raspberry-pi-images/85135#85135 RPI3 specific

  • https://github.com/bztsrc/raspi3-tutorial, getting started: https://raspberrypi.stackexchange.com/questions/34733/how-to-do-qemu-emulation-for-bare-metal-raspberry-pi-images/85135#85135

  • https://github.com/dwelch67/raspberrypi

  • https://github.com/BrianSidebotham/arm-tutorial-rpi

• gem5:

  • https://github.com/tukl-msd/gem5.bare-metal bare metal UART example. Tested with: https://stackoverflow.com/questions/43682311/uart-communication-in-gem5-with-arm-bare-metal/50983650#50983650

• games:

  • https://github.com/kcsongor/arm-doom PI 1 model B https://www.youtube.com/watch?v=jeHtktKtGYQ

  • https://github.com/Tetris-Duel-Team/Tetris-Duel Demo: https://www.youtube.com/watch?v=hTqKRdcKZ9k

  • https://github.com/ICTeam28/PiFox rail shooter https://www.youtube.com/watch?v=-5n9IxSQH1M

x86 bare metal tutorial at: https://github.com/cirosantilli/x86-bare-metal-examples