lkl.txt

Introduction
============

LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code as
extensively as possible with minimal effort and reduced maintenance overhead.

Examples of how LKL can be used are: creating userspace applications (running on
Linux and other operating systems) that can read or write Linux filesystems or
can use the Linux networking stack, creating kernel drivers for other operating
systems that can read Linux filesystems, bootloaders support for reading/writing
Linux filesystems, etc.

With LKL, the kernel code is compiled into an object file that can be directly
linked by applications. The API offered by LKL is based on the Linux system call
interface.

LKL is implemented as an architecture port in arch/lkl. It uses host operations
defined by the application or a host library (tools/lkl/lib).


Supported hosts
===============

The supported hosts for now are POSIX and Windows userspace applications.


Building LKL, the host library and LKL based tools
==================================================

    $ make -C tools/lkl

will build LKL as a object file, it will install it in tools/lkl/lib together
with the headers files in tools/lkl/include then will build the host library,
tests and a few of application examples:

* tests/boot - a simple applications that uses LKL and exercises the basic LKL
APIs

* fs2tar - a tool that converts a filesystem image to a tar archive

* cptofs/cpfromfs - a tool that copies files to/from a filesystem image

* lklfuse - a tool that can mount a filesystem image in userspace,
  without root priviledges, using FUSE


Building LKL on FreeBSD
-----------------------

    $ pkg install binutils gcc gnubc gmake gsed coreutils bison flex python argp-standalone

    #Prefer ports binutils and GNU bc(1):
    $ export PATH=/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/lib64/ccache

    $ gmake -C tools/lkl

Building LKL on Ubuntu
-----------------------

    $ sudo apt-get install libfuse-dev libarchive-dev xfsprogs libjsmn-dev
    
    # Optional, if you would like to be able to run tests
    $ sudo apt-get install btrfs-tools
    $ pip install yamlish junit_xml
    
    $ make -C tools/lkl
    
    # To check that everything works:
    $ cd tools/lkl
    $ make run-tests


Building LKL for Windows
------------------------

In order to build LKL for Win32 the mingw cross compiler needs to be installed
on the host (e.g. on Ubuntu the following packages are required:
binutils-mingw-w64-i686, gcc-mingw-w64-base, gcc-mingw-w64-i686
mingw-w64-common, mingw-w64-i686-dev).

Due to a bug in mingw regarding weak symbols the following patches needs to be
applied to mingw-binutils:

https://sourceware.org/ml/binutils/2015-10/msg00234.html

and i686-w64-mingw32-gas, i686-w64-mingw32-ld and i686-w64-mingw32-objcopy need
to be recompiled.

With that pre-requisites fullfilled you can now build LKL for Win32 with the
following command:

    $ make CROSS_COMPILE=i686-w64-mingw32- -C tools/lkl

    
    
Building LKL on Windows
------------------------

To build on Windows, certain GNU tools need to be installed. These tools can come
from several different projects, such as cygwin, unxutils, gnu-win32 or busybox-w32. 
Below is one minimal/modular set-up based on msys2.

### Common build dependencies:
* [MSYS2](https://sourceforge.net/projects/msys2/) (provides GNU bash and many other utilities)
* Extra utilities from MSYS2/pacman: bc, base-devel 

### General considerations:
* No spaces in pathnames (source, prefix, destination,...)! 
* Make sure that all utilities are in the PATH.  
* Win64 (and MinGW 64-bit crt) is LLP64, which causes conflicts in size of "long" in the 
Linux source. Linux (and lkl) can (currently) not 
be built on LLP64.
* Cygwin (and msys2) are LP64, like linux. 

### For MSYS2 (and Cygwin):
Msys2 will install a gcc tool chain as part of the base-devel bundle. Binutils (2.26) is already 
patched for NT weak externals. Using the msys2 shell, cd to the lkl sources and run:

    $ make -C tools/lkl

### For MinGW:
Install mingw-w64-i686-toolchain via pacman, mingw-w64-i686-binutils (2.26) is already patched
for NT weak externals. Start a MinGW Win32 shell (64-bit will not work, see above)
and run:

    $ make -C tools/lkl

LKL kernel configure
==================

You can configure LKL kernel just like you configured linux kernel source code by setting ARCH=lkl.
For example,the default configuration in defconfig may set "CONFIG_DEBUG_INFO=y".This option will
increase output size to about 130M. This is not negligible in some resource-constrained envirmonments.

    $ make ARCH=lkl menuconfig

Set Kernel hacking --->Compile-time checks and compiler options --->Debug information
(Disable debug information),this should shrink size to about 15M.

    $ make clean -C tools/lkl
    $ make -C tools/lkl


As external lib/header detection only happens at a first build if the compilation environment changed
you may need to clean the configuration of build.

    $ make clean-conf -C tools/lkl

This will also remove your configuration in .config,backup it as needed.

LKL hijack library
==================

LKL hijack library (liblkl-hijack.so) is used to replace system calls used by an
application on the fly so that the application can use LKL instead of the kernel
of host operating system. LD_PRELOAD is used to dynamically override system
calls with this library when you execute a program.

You can usually use this library via a wrapper script.

    $ cd tools/lkl
    $ ./bin/lkl-hijack.sh ip address show

In order to configure the behavior of LKL, a json file can be used. You can
specify json file with environmental variables (LKL_HIJACK_CONFIG_FILE). If
there is nothing specified, LKL tries to find with the name 'lkl-hijack.json'
for the configuration file.  You can also use the old-style configuration with
environmental variables (e.g., LKL_HIJACK_NET_IFTYPE) but those are overridden
if a json file is specified.

```
     $ cat conf.json
     {
       "gateway":"192.168.0.1",
       "gateway6":"2001:db8:0:f101::1",
       "debug":"1",
       "singlecpu":"1",
       "sysctl":"net.ipv4.tcp_wmem=4096 87380 2147483647",
       "boot_cmdline":"ip=dhcp",
       "interfaces":[
               {
                       "mac":"12:34:56:78:9a:bc",
                       "type":"tap",
                       "param":"tap7",
                       "ip":"192.168.0.2",
                       "masklen":"24",
                       "ifgateway":"192.168.0.1",
                       "ipv6":"2001:db8:0:f101::2",
                       "masklen6":"64",
                       "ifgateway6":"2001:db8:0:f101::1",
                       "offload":"0xc803"
               },
               {
                       "mac":"12:34:56:78:9a:bd",
                       "type":"tap",
                       "param":"tap77",
                       "ip":"192.168.1.2",
                       "masklen":"24",
                       "ifgateway":"192.168.1.1",
                       "ipv6":"2001:db8:0:f102::2",
                       "masklen6":"64",
                       "ifgateway6":"2001:db8:0:f102::1",
                       "offload":"0xc803"
               }
       ]
     }
     $ LKL_HIJACK_CONFIG_FILE="conf.json" lkl-hijack.sh ip addr s
```

The following are the list of keys to describe a JSON file.

* IPv4 gateway address

  key: "gateway"
  value type: string

  the gateway IPv4 address of LKL network stack.
```
     "gateway":"192.168.0.1"
```

* IPv6 gateway address

  key: "gateway6"
  value type: string

  the gateway IPv6 address of LKL network stack.
```
     "gateway6":"2001:db8:0:f101::1"
```

* Debug

  key: "debug"
  value type: string

  Setting it causes some debug information (both from the kernel and the
  LKL library) to be enabled.  If zero' is specified it is disabled.
  It is also used as a bit mask to turn on specific debugging facilities.
  E.g., setting it to "0x100" will cause the LKL kernel to pause after
  the hijack'ed app exits. This allows one to debug or collect info from
  the LKL kernel before it quits.
```
     "debug":"1"
```

* Single CPU pinning

  key: "singlecpu"
  value type: string

  Pin LKL kernel threads on to a single host cpu. value "1" pins
  only LKL kernel threads while value "2" also pins polling
  threads.
```
     "singlecpu":"1"
```

* SYSCTL

  key: "sysctl"
  value type: string

  Configure sysctl values of the booted kernel via the hijack library. Multiple
  entries can be specified.
```
     "sysctl":"net.ipv4.tcp_wmem=4096 87380 2147483647"
```

* Boot command line

  key: "boot_cmdline"
  value type: string

  Specify the command line to the kernel boot so that change the configuration
  on a kernel instance.  For instance, you can change the memory size with
  below.
```
     "boot_cmdline": "mem=1G"
```

* Mount

  key: "mount"
  value type: string

```
     "mount": "proc,sysfs"
```

* Network Interface Configuration

  key: "interfaces"
  value type: array of objects

  This key takes a set of sub-keys to configure a single interface. Each key is defined as follows.
  ```
       "interfaces":[{....},{....}]
  ```


	* Interface type

	  key: "type"
	  value type: string

	  The interface type in host operating system to connect to LKL.
	  The following example specifies a tap interface.
	```
	     "type":"tap"
	```

	* Interface parameter

	  key: "param"
	  value type: string

	  Additional configuration parameters for the interface specified by Interface type (type).
	  The parameters depend on the interface type.
	```
	     "type":"tap",
	     "param":"tap0"
	```

	* Interface MTU size

	  key: "mtu"
	  value type: string

	  the MTU size of the interface.
	```
	     "mtu":"1280"
	```

	* Interface IPv4 address

	  key: "ip"
	  value type: string

	  the IPv4 address of the interface.
	  If you want to use DHCP for the IP address assignment,
	  use "boot_cmdline" with "ip=dhcp" option.
	```
	     "ip":"192.168.0.2"
	```
	```
	     "boot_cmdline":"ip=dhcp"
	```

	* Interface IPv4 netmask length

	  key: "masklen"
	  value type: string

	  the network mask length of the interface.
	```
	     "ip":"192.168.0.2",
	     "masklen":"24"
	```

	* Interface IPv4 gateway on routing policy table

	  key: "ifgateway"
	  value type: string

	  If you specify this parameter, LKL adds routing policy table.
	  And then LKL creates link local and gateway route on this table.
	  Table SELECTOR is "from" and PREFIX is address you assigned to this interface.
	  Table id is 2 * (interface index).
	  This parameter could be used to configure LKL for mptcp, for example.

	```
	     "ip":"192.168.0.2",
	     "masklen":"24",
	     "ifgateway":"192.168.0.1"
	```

	* Interface IPv6 address

	  key: "ipv6"
	  value type: string

	  the IPv6 address of the interface.
	```
	     "ipv6":"2001:db8:0:f101::2"
	```

	* Interface IPv6 netmask length

	  key: "masklen6"
	  value type: string

	  the network mask length of the interface.
	```
	     "ipv6":"2001:db8:0:f101::2",
	     "masklen":"64"
	```

	* Interface IPv6 gateway on routing policy table

	  key: "ifgateway6"
	  value type: string

	  If you specify this parameter, LKL adds routing policy table.
	  And then LKL creates link local and gateway route on this table.
	  Table SELECTOR is "from" and PREFIX is address you assigned to this interface.
	  Table id is 2 * (interface index) + 1.
	  This parameter could be used to configure LKL for mptcp, for example.
	```
	     "ipv6":"2001:db8:0:f101::2",
	     "masklen":"64"
	     "ifgateway6":"2001:db8:0:f101::1",
	```

	* Interface MAC address

	  key: "mac"
	  value type: string

	  the MAC address of the interface.
	```
	     "mac":"12:34:56:78:9a:bc"
	```

	* Interfac neighbor entries

	  key: "neigh"
	  value type: string

	  Add a list of permanent neighbor entries in the form of "ip|mac;ip|mac;...". ipv6 are supported
	```
	     "neigh":"192.168.0.1|12:34:56:78:9a:bc;2001:db8:0:f101::1|12:34:56:78:9a:be"
	```

	* Interface qdisc entries

	  key: "qdisc"
	  value type: string

	  Add a qdisc entry in the form of "root|type;root|type;...".
	```
	     "qdisc":"root|fq"
	```

	* Interface offload

	  key: "offload"
	  value type: string

	  Work as a bit mask to enable selective device offload features. E.g.,
	  to enable "mergeable RX buffer" (LKL_VIRTIO_NET_F_MRG_RXBUF) +
	  "guest csum" (LKL_VIRTIO_NET_F_GUEST_CSUM) device features, simply set
	  it to 0x8002.
	  See virtio_net.h for a list of offload features and their bit masks.
	```
	     "offload":"0x8002"
	```

* Delay

  key: "delay_main"
  value type: string

  The delay before calling main() function of the application after the
  initialization of LKL.  Some subsystems in Linux tree require a certain
  amount of time before accepting a request from application, such as
  delivery of address assignment to an network interface.  This parameter
  is used in such case.  The value is described as a microsecond value.
```
     "delay_main":"500000"
```

* nameserver

  key: "nameserver"
  value type: string

  a name server address, which will be written in /etc/resolv.conf into a
  filesystem used by a LKL instance.
```
     "nameserver":"8.8.8.8"
```

LKL hijack library with zpoline
-------------------------------

[zpoline](https://github.com/yasukata/zpoline) is an alternative to
syscall hijack based on LD_PRELOAD, which is still default on LKL.
The zpoline library works with binary rewrites to the loaded programs
upon instantiation, then load hook function for the original syscalls.
The LKL hijack library works together with zpoline by loading LKL.

zpoline currently only works on x86_64 machines.

To use the zpoline-enabled hijack library, please follow the
instruction below.

- Build
```
make -C tools/lkl -j8 zpoline=../zpoline
```

Suppose `zpoline` is downloaded at `../zpoline` and already build
before LKL build.

- Execution

zpoline rewrites the memory address 0x0 to hook syscalls, but non-root
users don't have a privilege to operate that address.  The following
configuration allows us to use zpoline without root privilege.

```
sudo sh -c "echo 0 > /proc/sys/vm/mmap_min_addr"
```

then, execute command with the environment variable `LKL_HIJACK_ZPOLINE=1`.

```
LKL_HIJACK_ZPOLINE=1 LKL_HIJACK_CONFIG_FILE=lkl-tap.json \
  ./tools/lkl/bin/lkl-hijack.sh ping www.google.com
```

The file `lkl-tap.json` can be prepared like this.

```
{
    "gateway": "172.17.0.1",
    "nameserver": "8.8.8.8",
    "interfaces": [
        {
            "ip": "172.17.0.39",
            "masklen": "16",
            "mac": "00:0d:0b:94:4e:97",
            "param": "tap0",
            "type": "tap"
        }
    ],
}
```

With the preload hijack library, which is the default one, it uses the
host name resolver and if the host uses a nameserver, defined at
`/etc/resolv.conf`, like 127.0.0.53, is not accepting DNS requests, in
a view of the LKL instance.

But with zpoline, it can successfully replace all syscalls for name
resolution so can `ping` with a name.

FAQ
===

Q: How is LKL different from UML?

A: UML prodivides a full OS environment (e.g. user/kernel separation, user
processes) and also has requirements (a filesystem, processes, etc.) that makes
it hard to use it for standalone applications. UML also relies heavily on Linux
hosts. On the other hand LKL is designed to be linked directly with the
application and hence does not have user/kernel separation which makes it easier
to use it in standalone applications.


Q: How is LKL different from LibOS?

A: LibOS re-implements high-level kernel APIs for timers, softirqs, scheduling,
sysctl, SLAB/SLUB, etc. LKL behaves like any arch port, implementing the arch
level operations requested by the Linux kernel. LKL also offers a host interface
so that support for multiple hosts can be implemented.