massive doc updates

* examples/kubernetes: newly added
* docs/rootless.md: cleaned up for better readability
* examples/README.md: split out from the main README.md
* examples/build-using-dockerfile/README.md: split out from the main README.md
* README.md: add TOC using https://github.com/thlorenz/doctoc
* README.md: add mTLS configuration (relates to moby#1074)
* README.md: add more adoptions
* README.md: add inline cache (fix moby#976)

Signed-off-by: Akihiro Suda <[email protected]>

Author: AkihiroSuda

.dockerignore

@@ -1,2 +1,3 @@
 bin
+.certs
 .tmp

.github/CONTRIBUTING.md

@@ -61,6 +61,11 @@ make && sudo make install
 
 You can also use `make binaries-all` to prepare `buildkitd.containerd_only` and `buildkitd.oci_only`.
 
+To build containerized `moby/buildkit:local` and `moby/buildkit:local-rootless` images:
+```bash
+make images
+```
+
 ### Run the unit- and integration-tests
 
 Running tests:

.gitignore

@@ -1,3 +1,4 @@
 bin
+release-out
+.certs
 .tmp
-release-out

README.md

@@ -1,81 +1,77 @@
 # Rootless mode (Experimental)
 
-Requirements:
-- runc `a00bf0190895aa465a5fbed0268888e2c8ddfe85` (Oct 15, 2018) or later
-- Some distros such as Debian (excluding Ubuntu) and Arch Linux require `sudo sh -c "echo 1 > /proc/sys/kernel/unprivileged_userns_clone"`.
-- RHEL/CentOS 7 requires `sudo sh -c "echo 28633 > /proc/sys/user/max_user_namespaces"`. You may also need `sudo grubby --args="namespace.unpriv_enable=1 user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"`.
-- `newuidmap` and `newgidmap` need to be installed on the host. These commands are provided by the `uidmap` package. For RHEL/CentOS 7, RPM is not officially provided but available at https://copr.fedorainfracloud.org/coprs/vbatts/shadow-utils-newxidmap/ .
-- `/etc/subuid` and `/etc/subgid` should contain >= 65536 sub-IDs. e.g. `penguin:231072:65536`.
+Rootless mode allows running BuildKit daemon as a non-root user.
 
+## Distribution-specific hint
+Using Ubuntu kernel is recommended.
 
-## Set up
+### Ubuntu
+* No preparation is needed.
+* `overlayfs` snapshotter is enabled by default ([Ubuntu-specific kernel patch](https://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/commit/fs/overlayfs?id=3b7da90f28fe1ed4b79ef2d994c81efbc58f1144)).
 
-Setting up rootless mode also requires some bothersome steps as follows, but you can also use [`rootlesskit`](https://github.com/rootless-containers/rootlesskit) for automating these steps.
+### Debian GNU/Linux
+* Add `kernel.unprivileged_userns_clone=1` to `/etc/sysctl.conf` (or `/etc/sysctl.d`) and run `sudo sysctl -p`
+* To use `overlayfs` snapshotter (recommended), run `sudo modprobe overlay permit_mounts_in_userns=1` ([Debian-specific kernel patch, introduced in Debian 10](https://salsa.debian.org/kernel-team/linux/blob/283390e7feb21b47779b48e0c8eb0cc409d2c815/debian/patches/debian/overlayfs-permit-mounts-in-userns.patch)). Put the configuration to `/etc/modprobe.d` for persistence.
 
-### Terminal 1:
+### Arch Linux
+* Add `kernel.unprivileged_userns_clone=1` to `/etc/sysctl.conf` (or `/etc/sysctl.d`) and run `sudo sysctl -p`
 
-```
-$ unshare -U -m
-unshared$ echo $$ > /tmp/pid
-```
+### Fedora 31 and later
+* Run `sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"` and reboot.
 
-Unsharing mountns (and userns) is required for mounting filesystems without real root privileges.
+### Fedora 30
+* No preparation is needed
 
-### Terminal 2:
+### RHEL/CentOS 8
+* No preparation is needed
 
-```
-$ id -u
-1001
-$ grep $(whoami) /etc/subuid
-penguin:231072:65536
-$ grep $(whoami) /etc/subgid
-penguin:231072:65536
-$ newuidmap $(cat /tmp/pid) 0 1001 1 1 231072 65536
-$ newgidmap $(cat /tmp/pid) 0 1001 1 1 231072 65536
-```
+### RHEL/CentOS 7
+* Add `user.max_user_namespaces=28633` to `/etc/sysctl.conf` (or `/etc/sysctl.d`) and run `sudo sysctl -p`
+* Old releases (<= 7.6) require [extra configuration steps](https://github.com/moby/moby/pull/40076).
 
-### Terminal 1:
+### Container-Optimized OS from Google
+* :warning: Currently unsupported. See [#879](https://github.com/moby/buildkit/issues/879).
 
-```
-unshared# buildkitd
-```
+## Known limitations
+* No support for `overlayfs` snapshotter, except on Ubuntu and Debian kernels. We are planning to support `fuse-overlayfs` snapshotter instead for other kernels.
+* Network mode is always set to `network.host`.
+* No support for `containerd` worker
 
-* The data dir will be set to `/home/penguin/.local/share/buildkit`
-* The address will be set to `unix:///run/user/1001/buildkit/buildkitd.sock`
-* `overlayfs` snapshotter is not supported except Ubuntu-flavored kernel: http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/commit/fs/overlayfs?h=Ubuntu-4.13.0-25.29&id=0a414bdc3d01f3b61ed86cfe3ce8b63a9240eba7
-* containerd worker is not supported ( pending PR: https://github.com/containerd/containerd/pull/2006 )
-* Network namespace is not used at the moment.
-* Cgroups is disabled.
+## Running BuildKit in Rootless mode
 
-### Terminal 2:
+[RootlessKit](https://github.com/rootless-containers/rootlesskit/) needs to be installed.
 
+```console
+$ rootlesskit buildkitd
 ```
-$ go get ./examples/build-using-dockerfile
-$ build-using-dockerfile --buildkit-addr unix:///run/user/1001/buildkit/buildkitd.sock -t foo /path/to/somewhere
+
+```console
+$ buildctl --addr unix:///run/user/$UID/buildkit/buildkitd.sock build ...
 ```
 
-## Set up (using a container)
+## Containerized deployment
 
-Docker image is available as [`moby/buildkit:rootless`](https://hub.docker.com/r/moby/buildkit/tags/).
+### Kubernetes
+See [`../examples/kubernetes`](../examples/kubernetes).
 
-```
-$ docker run --name buildkitd -d --privileged -p 1234:1234  moby/buildkit:rootless --addr tcp://0.0.0.0:1234
-```
+### Docker
 
+```console
+$ docker run --name buildkitd -d --security-opt seccomp=unconfined --security-opt apparmor=unconfined moby/buildkit:rootless --oci-worker-no-process-sandbox
+$ buildctl --addr docker-container://buildkitd build ...
 ```
-$ go get ./examples/build-using-dockerfile
-$ build-using-dockerfile --buildkit-addr tcp://127.0.0.1:1234 -t foo /path/to/somewhere
-```
+#### About `--oci-worker-no-process-sandbox`
 
-### Security consideration
+By adding `--oci-worker-no-process-sandbox` to the `buildkitd` arguments, BuildKit can be executed in a container without adding `--privileged` to `docker run` arguments.
+However, you still need to pass `--security-opt seccomp=unconfined --security-opt apparmor=unconfined` to `docker run`.
+
+Note that `--oci-worker-no-process-sandbox` allows build executor containers to `kill` (and potentially `ptrace` depending on the seccomp configuration) an arbitrary process in the BuildKit daemon container.
 
-Although `moby/buildkit:rootless` executes the BuildKit daemon as a normal user, `docker run` still requires `--privileged`.
-This is to allow build executor containers to mount `/proc`, by providing "unmasked" `/proc` to the BuildKit daemon container.
+To allow running rootless `buildkitd` without `--oci-worker-no-process-sandbox`, run `docker run` with `--security-opt systempaths=unconfined`. (For Kubernetes, set `securityContext.procMount` to `Unmasked`.)
 
-See [`docker/cli#1347`](https://github.com/docker/cli/pull/1347) for the ongoing work to remove this requirement.
-See also [Disabling process sandbox](#disabling-process-sandbox).
+The `--security-opt systempaths=unconfined` flag disables the masks for the `/proc` mount in the container and potentially allows reading and writing dangerous kernel files, but it is safe when you are running `buildkitd` as non-root.
 
-#### UID/GID
+### Change UID/GID
 
 The `moby/buildkit:rootless` image has the following UID/GID configuration:
 
@@ -101,85 +97,8 @@ user:100000:65536
 
 To change the UID/GID configuration, you need to modify and build the BuildKit image manually.
 ```
-$ vi hack/dockerfiles/test.Dockerfile
-$ docker build -t buildkit-rootless-custom --target rootless -f hack/dockerfiles/test.Dockerfile .
-```
-
-#### Disabling process sandbox
-
-By passing `--oci-worker-no-process-sandbox` to the `buildkitd` arguments, BuildKit can be executed in a container without `--privileged`.
-However, you still need to pass `--security-opt seccomp=unconfined --security-opt apparmor=unconfined` to `docker run`.
-
-```
-$ docker run --name buildkitd -d --security-opt seccomp=unconfined --security-opt apparmor=unconfined -p 1234:1234 moby/buildkit:rootless --addr tcp://0.0.0.0:1234 --oci-worker-no-process-sandbox
-```
-
-Note that `--oci-worker-no-process-sandbox` allows build executor containers to `kill` (and potentially `ptrace` depending on the seccomp configuration) an arbitrary process in the BuildKit daemon container.
-
-
-## Set up (using Kubernetes)
-
-### With `securityContext`
-
-```yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  labels:
-    app: buildkitd
-  name: buildkitd
-spec:
-  selector:
-    matchLabels:
-      app: buildkitd
-  template:
-    metadata:
-      labels:
-        app: buildkitd
-    spec:
-      containers:
-      - image: moby/buildkit:rootless
-        args:
-        - --addr
-        - tcp://0.0.0.0:1234
-        name: buildkitd
-        ports:
-        - containerPort: 1234
-        securityContext:
-          privileged: true
-```
-
-This configuration requires privileged containers to be enabled.
-
-If you are using Kubernetes v1.12+ with either Docker v18.06+, containerd v1.2+, or CRI-O v1.12+ as the CRI runtime, you can replace `privileged: true` with `procMount: Unmasked`.
-
-### Without `securityContext` but with `--oci-worker-no-process-sandbox`
-
-```yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  labels:
-    app: buildkitd
-  name: buildkitd
-spec:
-  selector:
-    matchLabels:
-      app: buildkitd
-  template:
-    metadata:
-      labels:
-        app: buildkitd
-    spec:
-      containers:
-      - image: moby/buildkit:rootless
-        args:
-        - --addr
-        - tcp://0.0.0.0:1234
-        - --oci-worker-no-process-sandbox
-        name: buildkitd
-        ports:
-        - containerPort: 1234
+$ vi hack/dockerfiles/test.buildkit.Dockerfile
+$ make images
+$ docker run ... moby/buildkit:local-rootless ...
 ```
 
-See [Disabling process sandbox](#disabling-process-sandbox) for security notice.

docs/rootless.md

@@ -0,0 +1,39 @@
+# BuildKit Examples
+
+## Kubernetes manifests
+- [`./kubernetes`](./kubernetes): Kubernetes manifests (`Pod`, `Deployment`, `StatefulSet`, `Job`)
+
+## CLI examples
+- [`./buildctl-daemonless`](./buildctl-daemonless): buildctl without daemon
+- [`./build-using-dockerfile`](./build-using-dockerfile): an example BuildKit client with `docker build`-style CLI
+
+## LLB examples
+
+For understanding the basics of LLB, `buildkit*` directory contains scripts that define how to build different configurations of BuildKit itself and its dependencies using the `client` package. Running one of these scripts generates a protobuf definition of a build graph. Note that the script itself does not execute any steps of the build.
+
+You can use `buildctl debug dump-llb` to see what data is in this definition. Add `--dot` to generate dot layout.
+
+```bash
+go run examples/buildkit0/buildkit.go \
+    | buildctl debug dump-llb \
+    | jq .
+```
+
+To start building use `buildctl build` command. The example script accepts `--with-containerd` flag to choose if containerd binaries and support should be included in the end result as well.
+
+```bash
+go run examples/buildkit0/buildkit.go \
+    | buildctl build
+```
+
+`buildctl build` will show interactive progress bar by default while the build job is running. If the path to the trace file is specified, the trace file generated will contain all information about the timing of the individual steps and logs.
+
+Different versions of the example scripts show different ways of describing the build definition for this project to show the capabilities of the library. New versions have been added when new features have become available.
+
+-  `./buildkit0` - uses only exec operations, defines a full stage per component.
+-  `./buildkit1` - cloning git repositories has been separated for extra concurrency.
+-  `./buildkit2` - uses git sources directly instead of running `git clone`, allowing better performance and much safer caching.
+-  `./buildkit3` - allows using local source files for separate components eg. `./buildkit3 --runc=local | buildctl build --local runc-src=some/local/path`
+-  `./dockerfile2llb` - can be used to convert a Dockerfile to LLB for debugging purposes
+-  `./nested-llb` - shows how to use nested invocation to generate LLB
+-  `./gobuild` - shows how to use nested invocation to generate LLB for Go package internal dependencies

examples/README.md

@@ -0,0 +1,15 @@
+# `build-using-dockerfile` example
+
+:information_source: [BuildKit has been integrated to `docker build` since Docker 18.06.](https://docs.docker.com/develop/develop-images/build_enhancements/)
+The `build-using-dockerfile` CLI is just provided as an example for writing a BuildKit client application.
+
+For people familiar with `docker build` command, `build-using-dockerfile` is provided as an example for building Dockerfiles with BuildKit using a syntax similar to `docker build`.
+
+```bash
+go get .
+
+build-using-dockerfile -t myimage /path/to/dir
+
+# build-using-dockerfile will automatically load the resulting image to Docker
+docker inspect myimage
+```

examples/build-using-dockerfile/README.md

@@ -0,0 +1 @@
+kubernetes/consistenthash

examples/kube-consistent-hash

@@ -0,0 +1,79 @@
+# Kubernetes manifests for BuildKit
+
+This directory contains Kubernetes manifests for `Pod`, `Deployment` (with `Service`), `StatefulSet`, and `Job`.
+* `Pod`: good for quick-start
+* `Deployment` + `Service`: good for random load balancing with registry-side cache
+* `StateFulset`: good for client-side load balancing, without registry-side cache
+* `Job`: good if you don't want to have daemon pods
+
+Using Rootless mode (`*.rootless.yaml`) is recommended because Rootless mode image is executed as non-root user (UID 1000) and doesn't need `securityContext.privileged`.
+
+:warning: Rootless mode may not work on some host kernels. See [`../../docs/rootless.md`](../../docs/rootless.md).
+
+See also ["Building Images Efficiently And Securely On Kubernetes With BuildKit" (KubeCon EU 2019)](https://kccnceu19.sched.com/event/MPX5).
+
+## `Pod`
+
+```console
+$ kubectl apply -f pod.rootless.yaml
+$ buildctl \
+  --addr kube-pod://buildkitd \
+  build --frontend dockerfile.v0 --local context=/path/to/dir --local dockerfile=/path/to/dir
+```
+
+If rootless mode doesn't work, try `pod.privileged.yaml`.
+
+:warning: `kube-pod://` connection helper requires Kubernetes role that can access `pods/exec` resources. If `pods/exec` is not accessible, use `Service` instead (See below).
+
+## `Deployment` + `Service`
+
+Setting up mTLS is highly recommended.
+
+`./create-certs.sh SAN [SAN...]` can be used for creating certificates.
+```console
+$ ./create-certs.sh 127.0.0.1
+```
+
+The daemon certificates is created as `Secret` manifest named `buildkit-daemon-certs`.
+```console
+$ kubectl apply -f .certs/buildkit-daemon-certs.yaml
+```
+
+Apply the `Deployment` and `Service` manifest:
+```console
+$ kubectl apply -f deployment+service.rootless.yaml
+$ kubectl scale --replicas=10 deployment/buildkitd
+```
+
+Run `buildctl` with TLS client certificates:
+```console
+$ kubectl port-forward service/buildkitd 1234
+$ buildctl \
+  --addr tcp://127.0.0.1:1234 \
+  --tlscacert .certs/client/ca.pem \
+  --tlscert .certs/client/cert.pem \
+  --tlskey .certs/client/key.pem \
+  build --frontend dockerfile.v0 --local context=/path/to/dir --local dockerfile=/path/to/dir
+```
+
+## `StatefulSet`
+`StatefulSet` is useful for consistent hash mode.
+
+```console
+$ kubectl apply -f statefulset.rootless.yaml
+$ kubectl scale --replicas=10 statefulset/buildkitd
+$ buildctl \
+  --addr kube-pod://buildkitd-4 \
+  build --frontend dockerfile.v0 --local context=/path/to/dir --local dockerfile=/path/to/dir
+```
+
+See `[./consistenthash`](./consistenthash) for how to use consistent hashing.
+
+## `Job`
+
+```console
+$ kubectl apply -f job.rootless.yaml
+```
+
+To push the image to the registry, you also need to mount `~/.docker/config.json`
+and set `$DOCKER_CONFIG` to `/path/to/.docker` directory.

examples/kubernetes/README.md

@@ -5,7 +5,7 @@ Demo for efficiently using BuildKit daemon-local cache with multi-node cluster
 ## Deploy
 
 ```console
-$ kubectl apply -f buildkitd-rootless-statefulset.yaml
+$ kubectl apply -f ../statefulset.rootless.yaml
 $ kubectl scale --replicas=10 statefulset/buildkitd
 ```
 
@@ -19,7 +19,7 @@ For example, the key can be defined as `<REPO NAME>:<CONTEXT PATH>`, e.g.
 
 Then determine the pod that corresponds to the key:
 ```console
-$ go get ./consistenthash
+$ go build -o consistenthash .
 $ pod=$(./show-running-pods.sh | consistenthash $key)
 ```