Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
16adfdc
fix(ci): fix release-vm-kernel workflow failures
drew Apr 9, 2026
44f67ac
refactor(vm): build kernel once on Linux, reuse kernel.c on macOS
drew Apr 9, 2026
709bc44
fix(vm): fix CI kernel build failures
drew Apr 9, 2026
4df66b0
fix(vm): add zstd and jq to build dependencies
drew Apr 9, 2026
7f127e4
fix(vm): resolve kernel-dir to absolute path before cd
drew Apr 9, 2026
242e9f4
fix(vm): use mise for Rust on macOS to avoid LLVM conflicts
drew Apr 9, 2026
1b79bdb
fix(vm): use rustup directly on macOS, disable sccache
drew Apr 9, 2026
18a5c58
fix(ci): fix release-vm-dev pipeline failures
drew Apr 9, 2026
04076c5
fix(ci): add zstd to CI environment for rootfs compression
drew Apr 9, 2026
0e1d041
fix(ci): add zstd to build-vm-linux and build-vm-macos jobs
drew Apr 9, 2026
3a6bde0
feat(vm): add install-vm.sh one-liner for quick VM binary install
drew Apr 9, 2026
70e53fb
fix(vm): align install-vm.sh safeguards with install.sh
drew Apr 9, 2026
463eb2b
docs(vm): expand release notes with install-vm.sh usage
drew Apr 9, 2026
0765fa2
docs(vm): simplify release notes install section
drew Apr 9, 2026
3ffb051
fix(vm): allow release-assets.githubusercontent.com in origin validation
drew Apr 9, 2026
b559057
fix(vm): create libkrunfw.dylib symlink on macOS after extraction
drew Apr 9, 2026
de7e439
fix(vm): re-exec with DYLD_LIBRARY_PATH on macOS for libkrunfw discovery
drew Apr 9, 2026
4ebfa45
fix(vm): retry gvproxy port forwarding with exponential backoff
drew Apr 9, 2026
be30c76
fix(vm): kill stale gvproxy holding target ports when state file is m…
drew Apr 9, 2026
bbe30a2
chore(vm): fix rustfmt formatting
drew Apr 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 20 additions & 3 deletions .github/workflows/release-vm-dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,11 @@ jobs:
name: Download Kernel Runtime
runs-on: build-amd64
timeout-minutes: 10
container:
image: ghcr.io/nvidia/openshell/ci:latest
credentials:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v4

Expand Down Expand Up @@ -150,6 +155,9 @@ jobs:
- name: Install tools
run: mise install

- name: Install zstd
run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/*

- name: Build base rootfs tarball
run: |
set -euo pipefail
Expand Down Expand Up @@ -223,6 +231,9 @@ jobs:
cache-directories: .cache/sccache
cache-targets: "true"

- name: Install zstd
run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/*

- name: Download kernel runtime tarball
uses: actions/download-artifact@v4
with:
Expand Down Expand Up @@ -334,6 +345,9 @@ jobs:
- name: Set up Docker Buildx
uses: ./.github/actions/setup-buildx

- name: Install zstd
run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/*

- name: Download kernel runtime tarball
uses: actions/download-artifact@v4
with:
Expand Down Expand Up @@ -506,10 +520,13 @@ jobs:
| Linux x86_64 | `openshell-vm-x86_64-unknown-linux-gnu.tar.gz` |
| macOS ARM64 | `openshell-vm-aarch64-apple-darwin.tar.gz` |

**macOS users:** The binary must be codesigned with the Hypervisor entitlement:
```bash
codesign --entitlements crates/openshell-vm/entitlements.plist --force -s - ./openshell-vm
### Quick install

```
curl -fsSL https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install-vm.sh | sh
```

Auto-detects your platform, verifies checksums, and codesigns on macOS.

files: |
release-final/openshell-vm-aarch64-unknown-linux-gnu.tar.gz
Expand Down
60 changes: 41 additions & 19 deletions .github/workflows/release-vm-kernel.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ name: Release VM Kernel
# "vm-dev" GitHub Release and consumed by release-vm-dev.yml when building the
# openshell-vm binary.
#
# The Linux kernel is compiled once on aarch64 Linux. The resulting kernel.c
# (a C source file containing the kernel as a byte array) is shared with the
# macOS job, which only needs to compile it into a .dylib — no krunvm, no
# Fedora VM, no kernel rebuild. This cuts macOS CI from ~45 min to ~5 min.
#
# This workflow runs on-demand (or when kernel config / pins change). It is
# intentionally decoupled from the per-commit VM binary build because the
# kernel rarely changes and takes 15-45 minutes to compile.
Expand All @@ -14,6 +19,7 @@ on:

permissions:
contents: write
packages: read

# Serialize with release-vm-dev.yml — both update the vm-dev release.
concurrency:
Expand All @@ -26,7 +32,7 @@ defaults:

jobs:
# ---------------------------------------------------------------------------
# Linux ARM64 — native kernel + libkrun build
# Linux ARM64 — native kernel + libkrun build (also exports kernel.c)
# ---------------------------------------------------------------------------
build-runtime-linux-arm64:
name: Build Runtime (Linux ARM64)
Expand Down Expand Up @@ -56,13 +62,25 @@ jobs:
--build-dir target/libkrun-build \
--output artifacts/vm-runtime-linux-aarch64.tar.zst

- name: Upload artifact
- name: Upload runtime artifact
uses: actions/upload-artifact@v4
with:
name: vm-runtime-linux-arm64
path: artifacts/vm-runtime-linux-aarch64.tar.zst
retention-days: 5

# Export kernel.c + ABI_VERSION for the macOS job. kernel.c contains
# the aarch64 Linux kernel as a byte array — it is OS-agnostic and can
# be compiled into a .dylib by Apple's cc without rebuilding the kernel.
- name: Upload kernel.c for macOS build
uses: actions/upload-artifact@v4
with:
name: kernel-c-arm64
path: |
target/libkrun-build/kernel.c
target/libkrun-build/ABI_VERSION
retention-days: 1

# ---------------------------------------------------------------------------
# Linux AMD64 — native kernel + libkrun build
# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -102,34 +120,35 @@ jobs:
retention-days: 5

# ---------------------------------------------------------------------------
# macOS ARM64 — kernel built via krunvm, libkrun built natively
# macOS ARM64 — uses pre-built kernel.c from Linux ARM64 job
# ---------------------------------------------------------------------------
build-runtime-macos-arm64:
name: Build Runtime (macOS ARM64)
needs: [build-runtime-linux-arm64]
runs-on: macos-latest-xlarge
timeout-minutes: 90
timeout-minutes: 30
env:
RUSTC_WRAPPER: ""
steps:
- uses: actions/checkout@v4

- name: Install dependencies
run: |
set -euo pipefail
brew install rust lld dtc xz
# libkrunfw from Homebrew (used as a fallback/reference by build scripts)
brew install libkrunfw
# krunvm is needed to build the Linux kernel inside a Fedora VM
brew tap slp/krun
brew install krunvm
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
brew install lld dtc xz

- name: Build custom libkrunfw (kernel)
run: crates/openshell-vm/runtime/build-custom-libkrunfw.sh
- name: Download pre-built kernel.c
uses: actions/download-artifact@v4
with:
name: kernel-c-arm64
path: target/kernel-artifact

- name: Build portable libkrun
run: tasks/scripts/vm/build-libkrun-macos.sh
- name: Build libkrunfw + libkrun from pre-built kernel
run: tasks/scripts/vm/build-libkrun-macos.sh --kernel-dir target/kernel-artifact

- name: Package runtime tarball
env:
CUSTOM_PROVENANCE_DIR: target/custom-runtime
run: |
tasks/scripts/vm/package-vm-runtime.sh \
--platform darwin-aarch64 \
Expand Down Expand Up @@ -235,10 +254,13 @@ jobs:
| Linux x86_64 | `openshell-vm-x86_64-unknown-linux-gnu.tar.gz` |
| macOS ARM64 | `openshell-vm-aarch64-apple-darwin.tar.gz` |

**macOS users:** The binary must be codesigned with the Hypervisor entitlement:
```bash
codesign --entitlements crates/openshell-vm/entitlements.plist --force -s - ./openshell-vm
### Quick install

```
curl -fsSL https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install-vm.sh | sh
```

Auto-detects your platform, verifies checksums, and codesigns on macOS.

files: |
release/vm-runtime-linux-aarch64.tar.zst
Expand Down
40 changes: 24 additions & 16 deletions architecture/custom-vm-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,23 +116,29 @@ and makes it straightforward to correlate VM behavior with a specific runtime ar
```mermaid
graph LR
subgraph Source["crates/openshell-vm/runtime/"]
BUILD["build-custom-libkrunfw.sh\nClones libkrunfw, applies config, builds"]
KCONF["kernel/openshell.kconfig\nKernel config fragment"]
README["README.md\nOperator documentation"]
end

subgraph Output["target/custom-runtime/"]
LIB["libkrunfw.dylib\nCustom library"]
META["provenance.json\nBuild metadata"]
FRAG["openshell.kconfig\nConfig fragment used"]
FULL["kernel.config\nFull kernel .config"]
subgraph Linux["Linux CI (build-libkrun.sh)"]
BUILD_L["Build kernel + libkrunfw.so + libkrun.so"]
KERNELC["kernel.c\nKernel as C byte array"]
end

KCONF --> BUILD
BUILD --> LIB
BUILD --> META
BUILD --> FRAG
BUILD --> FULL
subgraph macOS["macOS CI (build-libkrun-macos.sh)"]
BUILD_M["Compile kernel.c -> libkrunfw.dylib\nBuild libkrun.dylib"]
end

subgraph Output["target/libkrun-build/"]
LIB_SO["libkrunfw.so + libkrun.so\n(Linux)"]
LIB_DY["libkrunfw.dylib + libkrun.dylib\n(macOS)"]
end

KCONF --> BUILD_L
BUILD_L --> LIB_SO
BUILD_L --> KERNELC
KERNELC --> BUILD_M
BUILD_M --> LIB_DY
```

## Kernel Config Fragment
Expand Down Expand Up @@ -222,16 +228,18 @@ supported platforms. Runs on-demand or when the kernel config / pinned versions

| Platform | Runner | Build Method |
|----------|--------|-------------|
| Linux ARM64 | `build-arm64` (self-hosted) | Native `build-libkrun.sh` |
| Linux ARM64 | `build-arm64` (self-hosted) | Native `build-libkrun.sh` (also exports kernel.c) |
| Linux x86_64 | `build-amd64` (self-hosted) | Native `build-libkrun.sh` |
| macOS ARM64 | `macos-latest-xlarge` (GitHub-hosted) | `build-custom-libkrunfw.sh` (krunvm) + `build-libkrun-macos.sh` |
| macOS ARM64 | `macos-latest-xlarge` (GitHub-hosted) | `build-libkrun-macos.sh --kernel-dir` (uses pre-built kernel.c from ARM64) |

Artifacts: `vm-runtime-{platform}.tar.zst` containing libkrun, libkrunfw, gvproxy, and
provenance metadata.

The macOS kernel build requires a real macOS ARM64 runner because it uses `krunvm` to
compile the Linux kernel inside a Fedora VM (Hypervisor.framework). The kernel inside
libkrunfw is always Linux regardless of host platform.
The aarch64 Linux kernel is compiled once on the Linux ARM64 runner. The resulting
`kernel.c` (a C source file containing the kernel as a byte array) is passed to the
macOS job, which compiles it into `libkrunfw.dylib` with Apple's `cc`. This eliminates
the need for krunvm/Fedora VM and cuts macOS CI from ~45 min to ~5 min. The kernel
inside libkrunfw is always Linux regardless of host platform.

### VM Binary (`release-vm-dev.yml`)

Expand Down
2 changes: 1 addition & 1 deletion crates/openshell-vm/pins.env
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# Pinned dependency versions for openshell-vm builds.
#
# This file is sourced by build-rootfs.sh and
# build-custom-libkrunfw.sh. It centralises version pins and content-addressed
# build-libkrun.sh. It centralises version pins and content-addressed
# digests so that builds are reproducible and auditable.
#
# Environment variables override these defaults — CI and local dev workflows
Expand Down
62 changes: 36 additions & 26 deletions crates/openshell-vm/runtime/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> Status: Experimental and work in progress (WIP). VM support is under active development and may change.

This directory contains the build infrastructure for a custom `libkrunfw` runtime
This directory contains the kernel config fragment for a custom `libkrunfw` runtime
that enables bridge CNI and netfilter support in the OpenShell gateway VM.

## Why
Expand All @@ -21,51 +21,61 @@ that enables these networking and sandboxing features.

```
runtime/
build-custom-libkrunfw.sh # Build script for custom libkrunfw
kernel/
openshell.kconfig # Kernel config fragment (networking + sandboxing)
```

## Building
## Build Pipeline

### Prerequisites
The kernel is compiled on Linux CI runners. macOS reuses the pre-built `kernel.c`
artifact from the Linux ARM64 build — no krunvm or Fedora VM needed.

- Rust toolchain
- make, git, curl
- On macOS: Xcode command line tools and cross-compilation tools for aarch64
```
Linux ARM64: builds aarch64 kernel -> .so + exports kernel.c (parallel)
Linux AMD64: builds x86_64 kernel -> .so (parallel)
macOS ARM64: reuses aarch64 kernel.c -> .dylib (depends on ARM64)
```

### Build Scripts

| Script | Platform | What it does |
|--------|----------|-------------|
| `tasks/scripts/vm/build-libkrun.sh` | Linux | Builds libkrunfw + libkrun from source, exports kernel.c |
| `tasks/scripts/vm/build-libkrun-macos.sh` | macOS | Compiles pre-built kernel.c into .dylib, builds libkrun |
| `tasks/scripts/vm/package-vm-runtime.sh` | Any | Packages runtime tarball (libs + gvproxy + provenance) |

### Quick Build
### Quick Build (Linux)

```bash
# Build custom libkrunfw (clones libkrunfw repo, applies config, builds)
./crates/openshell-vm/runtime/build-custom-libkrunfw.sh
# Build both libkrunfw and libkrun from source
tasks/scripts/vm/build-libkrun.sh

# Or build the full runtime from source via mise:
FROM_SOURCE=1 mise run vm:setup
```

### Output
### Quick Build (macOS)

Build artifacts are placed in `target/custom-runtime/`:
On macOS, you need a pre-built `kernel.c` from a Linux ARM64 build:

```
target/custom-runtime/
libkrunfw.dylib # The custom library
libkrunfw.<version>.dylib # Version-suffixed copy
provenance.json # Build metadata (commit, hash, timestamp)
openshell.kconfig # The config fragment used
kernel.config # Full kernel .config (for debugging)
```bash
# Download pre-built runtime (recommended, ~30s):
mise run vm:setup

# Or if you have kernel.c from a Linux build:
tasks/scripts/vm/build-libkrun-macos.sh --kernel-dir target/libkrun-build
```

### Using the Custom Runtime
### Output

```bash
# Point the bundle script at the custom build and rebuild:
export OPENSHELL_VM_RUNTIME_SOURCE_DIR=target/custom-runtime
mise run vm:build
Build artifacts are placed in `target/libkrun-build/`:

# Then boot the VM as usual:
mise run vm
```
target/libkrun-build/
libkrun.so / libkrun.dylib # The VMM library
libkrunfw.so* / libkrunfw.dylib # Kernel firmware library
kernel.c # Linux kernel as C byte array (Linux only)
ABI_VERSION # ABI version number (Linux only)
```

## Networking
Expand Down
Loading
Loading