Skip to content

chore(ci): add prebuilt z3 release pipeline #816

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
drew wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

chore(ci): add prebuilt z3 release pipeline #816

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 23 additions & 4 deletions .github/workflows/release-dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ permissions:
contents: write
packages: write

env:
Z3_RELEASE_TAG: z3-latest

defaults:
run:
shell: bash
Expand Down Expand Up @@ -129,8 +132,12 @@ jobs:
- name: Build Python wheels
run: |
set -euo pipefail
OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" mise run python:build:multiarch
OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" mise run python:build:macos
OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" \
OPENSHELL_Z3_RELEASE_TAG="${Z3_RELEASE_TAG}" \
mise run python:build:multiarch
OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" \
OPENSHELL_Z3_RELEASE_TAG="${Z3_RELEASE_TAG}" \
mise run python:build:macos
ls -la target/wheels/*.whl

- name: Capture wheel filenames
Expand All @@ -151,7 +158,8 @@ jobs:
# Build CLI binaries (Linux musl — static, native on each arch)
#
# Builds run directly on the CI host (glibc Ubuntu). Zig provides musl
# C/C++ toolchains for bundled-z3 and ring, and is also used as the linker.
# C/C++ toolchains for ring and the final musl linker. The downloaded Z3
# archives were also built with zig c++, so we keep libc++ for compatibility.
# ---------------------------------------------------------------------------
build-cli-linux:
name: Build CLI (Linux ${{ matrix.arch }})
Expand Down Expand Up @@ -234,6 +242,16 @@ jobs:
# Override z3-sys default (stdc++) so Rust links the matching runtime.
echo "CXXSTDLIB=c++" >> "$GITHUB_ENV"

- name: Download prebuilt Z3
run: |
set -euo pipefail
Z3_ROOT="$(sh tasks/scripts/z3/download-release-artifact.sh \
--target "${{ matrix.target }}" \
--release-tag "${Z3_RELEASE_TAG}" \
--output-dir "${RUNNER_TEMP:-/tmp}/prebuilt-z3")"
echo "PKG_CONFIG_ALLOW_CROSS=1" >> "$GITHUB_ENV"
echo "PKG_CONFIG_PATH=${Z3_ROOT}/lib/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}" >> "$GITHUB_ENV"

- name: Scope workspace to CLI crates
run: |
set -euo pipefail
Expand All @@ -246,7 +264,7 @@ jobs:
sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${{ needs.compute-versions.outputs.cargo_version }}"'"/}' Cargo.toml

- name: Build ${{ matrix.target }}
run: mise x -- cargo build --release --target ${{ matrix.target }} -p openshell-cli --features bundled-z3
run: mise x -- cargo build --release --target ${{ matrix.target }} -p openshell-cli

- name: sccache stats
if: always()
Expand Down Expand Up @@ -311,6 +329,7 @@ jobs:
--build-arg OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" \
--build-arg OPENSHELL_IMAGE_TAG=dev \
--build-arg CARGO_TARGET_CACHE_SCOPE="${{ github.sha }}" \
--build-arg Z3_RELEASE_TAG="${Z3_RELEASE_TAG}" \
--target binary \
--output type=local,dest=out/ \
.
Expand Down
25 changes: 21 additions & 4 deletions .github/workflows/release-tag.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ permissions:
# push-event ref. Every job references this via env.RELEASE_TAG.
env:
RELEASE_TAG: ${{ inputs.tag || github.ref_name }}
Z3_RELEASE_TAG: z3-latest

defaults:
run:
Expand Down Expand Up @@ -150,8 +151,12 @@ jobs:
- name: Build Python wheels
run: |
set -euo pipefail
OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" mise run python:build:multiarch
OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" mise run python:build:macos
OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" \
OPENSHELL_Z3_RELEASE_TAG="${Z3_RELEASE_TAG}" \
mise run python:build:multiarch
OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" \
OPENSHELL_Z3_RELEASE_TAG="${Z3_RELEASE_TAG}" \
mise run python:build:macos
ls -la target/wheels/*.whl

- name: Capture wheel filenames
Expand All @@ -172,7 +177,8 @@ jobs:
# Build CLI binaries (Linux musl — static, native on each arch)
#
# Builds run directly on the CI host (glibc Ubuntu). Zig provides musl
# C/C++ toolchains for bundled-z3 and ring, and is also used as the linker.
# C/C++ toolchains for ring and the final musl linker. The downloaded Z3
# archives were also built with zig c++, so we keep libc++ for compatibility.
# ---------------------------------------------------------------------------
build-cli-linux:
name: Build CLI (Linux ${{ matrix.arch }})
Expand Down Expand Up @@ -256,6 +262,16 @@ jobs:
# Override z3-sys default (stdc++) so Rust links the matching runtime.
echo "CXXSTDLIB=c++" >> "$GITHUB_ENV"

- name: Download prebuilt Z3
run: |
set -euo pipefail
Z3_ROOT="$(sh tasks/scripts/z3/download-release-artifact.sh \
--target "${{ matrix.target }}" \
--release-tag "${Z3_RELEASE_TAG}" \
--output-dir "${RUNNER_TEMP:-/tmp}/prebuilt-z3")"
echo "PKG_CONFIG_ALLOW_CROSS=1" >> "$GITHUB_ENV"
echo "PKG_CONFIG_PATH=${Z3_ROOT}/lib/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}" >> "$GITHUB_ENV"

- name: Scope workspace to CLI crates
run: |
set -euo pipefail
Expand All @@ -268,7 +284,7 @@ jobs:
sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${{ needs.compute-versions.outputs.cargo_version }}"'"/}' Cargo.toml

- name: Build ${{ matrix.target }}
run: mise x -- cargo build --release --target ${{ matrix.target }} -p openshell-cli --features bundled-z3
run: mise x -- cargo build --release --target ${{ matrix.target }} -p openshell-cli

- name: sccache stats
if: always()
Expand Down Expand Up @@ -334,6 +350,7 @@ jobs:
--build-arg OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" \
--build-arg OPENSHELL_IMAGE_TAG="${{ needs.compute-versions.outputs.semver }}" \
--build-arg CARGO_TARGET_CACHE_SCOPE="${{ github.sha }}" \
--build-arg Z3_RELEASE_TAG="${Z3_RELEASE_TAG}" \
--target binary \
--output type=local,dest=out/ \
.
Expand Down
225 changes: 225 additions & 0 deletions .github/workflows/release-z3.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,225 @@
name: Release Z3

on:
workflow_dispatch:
inputs:
z3_version:
description: "Z3 version to build (e.g. 4.16.0)"
required: true
type: string
prune_existing_assets:
description: "Delete previously published z3-latest assets before upload"
required: false
default: true
type: boolean
notes:
description: "Optional additional release notes"
required: false
type: string

permissions:
contents: write
packages: read

concurrency:
group: z3-latest-release
cancel-in-progress: false

env:
Z3_RELEASE_TAG: z3-latest

defaults:
run:
shell: bash

jobs:
build-z3-linux:
name: Build Z3 (${{ matrix.target }})
strategy:
matrix:
include:
- runner: build-amd64
target: x86_64-unknown-linux-musl
zig_target: x86_64-linux-musl
use_zig: true
- runner: build-arm64
target: aarch64-unknown-linux-musl
zig_target: aarch64-linux-musl
use_zig: true
- runner: build-amd64
target: x86_64-unknown-linux-gnu
use_zig: false
- runner: build-arm64
target: aarch64-unknown-linux-gnu
use_zig: false
runs-on: ${{ matrix.runner }}
timeout-minutes: 60
container:
image: ghcr.io/nvidia/openshell/ci:latest
credentials:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
options: --privileged
env:
MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v4

- name: Mark workspace safe for git
run: git config --global --add safe.directory "$GITHUB_WORKSPACE"

- name: Install tools
if: matrix.use_zig
run: mise install

- name: Set up zig wrappers
if: matrix.use_zig
run: |
set -euo pipefail
ZIG="$(mise which zig)"
mkdir -p /tmp/zig-z3
for tool in cc c++; do
printf '#!/bin/bash\nexec "%s" %s --target=%s "$@"\n' \
"$ZIG" "$tool" "${{ matrix.zig_target }}" > "/tmp/zig-z3/${tool}"
chmod +x "/tmp/zig-z3/${tool}"
done

- name: Build artifact
run: |
set -euo pipefail
mkdir -p artifacts
if [ "${{ matrix.use_zig }}" = "true" ]; then
export CC=/tmp/zig-z3/cc
export CXX=/tmp/zig-z3/c++
fi
sh tasks/scripts/z3/build-artifact.sh \
--target "${{ matrix.target }}" \
--z3-version "${{ inputs.z3_version }}" \
--output "artifacts/z3-${{ matrix.target }}.tar.gz"
ls -lh artifacts/

- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: z3-${{ matrix.target }}
path: artifacts/z3-${{ matrix.target }}.tar.gz
retention-days: 5

build-z3-macos:
name: Build Z3 (aarch64-apple-darwin)
runs-on: build-amd64
timeout-minutes: 60
container:
image: ghcr.io/nvidia/openshell/ci:latest
credentials:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
options: --privileged
volumes:
- /var/run/docker.sock:/var/run/docker.sock
steps:
- uses: actions/checkout@v4

- name: Mark workspace safe for git
run: git config --global --add safe.directory "$GITHUB_WORKSPACE"

- name: Set up Docker Buildx
uses: ./.github/actions/setup-buildx

- name: Build macOS artifact via Docker
run: |
set -euo pipefail
docker buildx build \
--file deploy/docker/Dockerfile.z3-macos \
--build-arg Z3_VERSION="${{ inputs.z3_version }}" \
--target artifact \
--output type=local,dest=artifacts/ \
.
ls -lh artifacts/

- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: z3-aarch64-apple-darwin
path: artifacts/z3-aarch64-apple-darwin.tar.gz
retention-days: 5

release-z3:
name: Release Z3
needs: [build-z3-linux, build-z3-macos]
runs-on: build-amd64
timeout-minutes: 10
steps:
- uses: actions/checkout@v4

- name: Download all Z3 artifacts
uses: actions/download-artifact@v4
with:
pattern: z3-*
path: release/
merge-multiple: true

- name: Generate checksums
run: |
set -euo pipefail
cd release
sha256sum z3-*.tar.gz > z3-checksums-sha256.txt
cat z3-checksums-sha256.txt

- name: Ensure z3-latest tag exists
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git tag -fa "${Z3_RELEASE_TAG}" -m "OpenShell Z3 Latest" "${GITHUB_SHA}"
git push --force origin "${Z3_RELEASE_TAG}"

- name: Prune stale Z3 assets from z3-latest release
if: inputs.prune_existing_assets
uses: actions/github-script@v7
with:
script: |
const [owner, repo] = process.env.GITHUB_REPOSITORY.split('/');
let release;
try {
release = await github.rest.repos.getReleaseByTag({ owner, repo, tag: 'z3-latest' });
} catch (err) {
if (err.status === 404) {
core.info('No existing z3-latest release; will create fresh.');
return;
}
throw err;
}
for (const asset of release.data.assets) {
if (asset.name.startsWith('z3-')) {
core.info(`Deleting stale asset: ${asset.name}`);
await github.rest.repos.deleteReleaseAsset({ owner, repo, asset_id: asset.id });
}
}

- name: Create / update z3-latest GitHub Release
uses: softprops/action-gh-release@v2
with:
name: OpenShell Z3 Latest
prerelease: true
tag_name: ${{ env.Z3_RELEASE_TAG }}
target_commitish: ${{ github.sha }}
body: |
Rolling prebuilt static Z3 artifacts for OpenShell release packaging.

Z3 version: `${{ inputs.z3_version }}`

Supported assets:
- `z3-x86_64-unknown-linux-musl.tar.gz`
- `z3-aarch64-unknown-linux-musl.tar.gz`
- `z3-x86_64-unknown-linux-gnu.tar.gz`
- `z3-aarch64-unknown-linux-gnu.tar.gz`
- `z3-aarch64-apple-darwin.tar.gz`

${{ inputs.notes }}
files: |
release/z3-x86_64-unknown-linux-musl.tar.gz
release/z3-aarch64-unknown-linux-musl.tar.gz
release/z3-x86_64-unknown-linux-gnu.tar.gz
release/z3-aarch64-unknown-linux-gnu.tar.gz
release/z3-aarch64-apple-darwin.tar.gz
release/z3-checksums-sha256.txt
7 changes: 6 additions & 1 deletion architecture/build-containers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,12 @@ openshell sandbox create --from <name>

This pulls `ghcr.io/nvidia/openshell-community/sandboxes/<name>:latest`.

## CLI and Wheel Release Builds

Standalone CLI archives and Python wheels do not compile Z3 from source during release packaging. A manually triggered GitHub Actions workflow, `release-z3.yml`, publishes target-specific static Z3 bundles to the rolling `z3-latest` GitHub release in `NVIDIA/OpenShell`.

Release workflows and packaging Dockerfiles download the matching archive for their target, verify it against `z3-checksums-sha256.txt`, and expose the bundled `z3.pc` via `PKG_CONFIG_PATH` so `z3-sys` links against the prebuilt static library. Local development can still opt into source-built Z3 with the existing `bundled-z3` feature when needed.

## Local Development

`mise run cluster` is the primary development command. It bootstraps a cluster if one doesn't exist, then performs incremental deploys for subsequent runs.
Expand Down Expand Up @@ -70,4 +76,3 @@ The harness runs isolated scenarios in temporary git worktrees, keeps its own st
- auto-detection checks for gateway-only, supervisor-only, shared, Helm-only, unrelated, and explicit-target changes
- cold vs warm rebuild comparisons for gateway and supervisor code changes
- container-ID invalidation coverage to verify gateway + Helm are retriggered when the cluster container changes

Loading
Loading