Containerize standalone tarball build (#2595)

Replaces #2443

Containerizes the process of building the standalone C library tarballs

The starting point for this PR is @cjnolet 's work in #2443, where he wrote this description:

> _This PR is based on @msarahan's original POC, with the logic of the ci sript moved into build.sh and invoked through the CI script. The readme is also moved into the build and install guide in the docs._

Changes relative to that PR:

* threads `PARALLEL_LEVEL` throught so builds are parallelized
* enables the build cluster (`sccache-dist`) in CI
* simplifies configuration flow (e.g. reduced duplication of default values, removal of unnecessary flexibility)
* makes flow of AWS creds for `sccache` more secure
* removes unnecessary configuration
* removes unnecessary package installs, consolidates package installs
* removes an unnecessary `git clone` of https://github.com/rapidsai/spdx-license-builder
* enables `sccache` in CI and with a pattern that could work locally (will share details privately)
* updates GitHub Actions third-party actions to their latest versions

## Notes for Reviewers

### How I tested this

Locally tried each of the new commands added to `build.md`, with `sccache` enabled.

Tried just `build.sh` without `sccache` enabled (that takes a lot longer to run).

<details><summary>code for flipping between those (click me)</summary>

```shell
# enabling sccache
export CI=true
# (private steps setting up 'sccache' profile)
AWS_ACCESS_KEY_ID=$(
  aws configure get aws_access_key_id \
    --profile sccache
)
AWS_SECRET_ACCESS_KEY=$(
  aws configure get aws_secret_access_key \
    --profile sccache
)
AWS_SESSION_TOKEN=$(
  aws configure get aws_session_token \
    --profile sccache
)
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN

# testing without 'sccache'
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN CI
```

</details>

```shell
clean() {
    sudo rm -rf ./{build,c/build/,cpp/build,libcuvs_c.tar.gz}
}

# all defaults
clean
./build.sh tarball

# customizing base image
clean
CUVS_TARBALL_CUDA_VERSION=12.9.2 \
CUVS_TARBALL_PYTHON_VERSION=3.11 \
  ./build.sh tarball

# customizing output directory, building tests 
clean
CUVS_TARBALL_BUILD_OUTPUT_DIR="${PWD}/dist" ./build.sh tarball --tarball-build-tests

# manual run (no build.sh)
clean

docker build \
  -f Dockerfile.standalone \
  --build-arg CUDA_VERSION="13.3.0" \
  --build-arg PYTHON_VERSION="3.14" \
  --build-arg RAPIDS_VERSION="$(head -1 ./VERSION | cut -d. -f1,2 )" \
  -t cuvs-standalone-c:local \
  .

mkdir -p "${PWD}/dist"
docker run --rm \
  -v "${PWD}:/workspace" \
  -v "${PWD}/dist:/build" \
  cuvs-standalone-c:local --tarball-build-tests
```

Saw high cache hit rates from `sccache` and everything working as expected.

Also looked at CI logs and saw that fully-cached jobs take around 30 minutes, pretty similar to the timings in CI today.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Mike Sarahan (https://github.com/msarahan)
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #2595

Author: jameslamb

.github/workflows/build.yaml

@@ -61,34 +61,54 @@ jobs:
       contents: read
     uses: rapidsai/shared-workflows/.github/workflows/compute-matrix.yaml@release/26.10
     with:
-      build_type: pull-request
+      build_type: ${{ inputs.build_type || 'branch' }}
       matrix_name: conda-cpp-build
   rocky8-clib-standalone-build:
-    needs: [build-details, rocky8-clib-standalone-build-matrix]
+    needs: [rocky8-clib-standalone-build-matrix]
+    runs-on: linux-${{ matrix.ARCH }}-cpu16
+    env:
+      RAPIDS_BUILD_TYPE: ${{ inputs.build_type || 'branch' }}
     permissions:
       actions: read
       contents: read
       id-token: write
       packages: read
       pull-requests: read
-    secrets: inherit # zizmor: ignore[secrets-inherit]
-    uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@release/26.10
     strategy:
       fail-fast: false
       matrix: ${{ fromJSON(needs.rocky8-clib-standalone-build-matrix.outputs.matrix) }}
-    with:
-      build_type: ${{ inputs.build_type || 'branch' }}
-      build-datetime: ${{ needs.build-details.outputs.build-datetime }}
-      branch: ${{ inputs.branch }}
-      arch: "${{matrix.ARCH}}"
-      date: ${{ inputs.date }}
-      container_image: "rapidsai/ci-wheel:26.10-cuda${{ matrix.CUDA_VER }}-${{ matrix.LINUX_VER }}-py${{ matrix.PY_VER }}"
-      node_type: "cpu16"
-      requires_license_builder: true
-      script: "ci/build_standalone_c.sh"
-      artifact-name: "libcuvs_c_${{ matrix.CUDA_VER }}_${{ matrix.ARCH }}.tar.gz"
-      file_to_upload: "libcuvs_c.tar.gz"
-      sha: ${{ inputs.sha }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          ref: ${{ inputs.sha || github.sha }}
+          fetch-depth: 0
+          persist-credentials: false
+      - name: Standardize repository information
+        uses: rapidsai/shared-actions/rapids-github-info@main
+        with:
+          branch: ${{ inputs.branch }}
+          date: ${{ inputs.date }}
+          sha: ${{ inputs.sha }}
+      - uses: aws-actions/configure-aws-credentials@cbe3b392738ccf3f987d68400dafcf4b0624a56c # v6.2.4
+        with:
+          role-to-assume: ${{ vars.AWS_ROLE_ARN }}
+          aws-region: ${{ vars.AWS_REGION }}
+          role-duration-seconds: 10800
+      - name: Setup sccache-dist
+        uses: rapidsai/shared-actions/setup-sccache-dist@main
+        with:
+          request-timeout: 7140
+      - name: Build standalone C tarball
+        env:
+          CUVS_TARBALL_CUDA_VERSION: ${{ matrix.CUDA_VER }}
+          CUVS_TARBALL_PYTHON_VERSION: ${{ matrix.PY_VER }}
+        run: |
+          ./build.sh tarball --tarball-build-tests
+      - name: Upload artifacts
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: libcuvs_c_${{ matrix.CUDA_VER }}_${{ matrix.ARCH }}.tar.gz
+          path: libcuvs_c.tar.gz
   rust-build-matrix:
     needs: cpp-build
     permissions:
@@ -128,7 +148,7 @@ jobs:
       contents: read
     uses: rapidsai/shared-workflows/.github/workflows/compute-matrix.yaml@release/26.10
     with:
-      build_type: pull-request
+      build_type: ${{ inputs.build_type || 'branch' }}
       matrix_name: conda-cpp-build
       matrix_filter: map(select(.ARCH == "amd64"))
   go-build:
@@ -161,7 +181,7 @@ jobs:
       contents: read
     uses: rapidsai/shared-workflows/.github/workflows/compute-matrix.yaml@release/26.10
     with:
-      build_type: pull-request
+      build_type: ${{ inputs.build_type || 'branch' }}
       matrix_name: conda-cpp-build
       matrix_filter: map(select(.ARCH == "amd64"))
   java-build:

.github/workflows/pr.yaml

@@ -563,30 +563,45 @@ jobs:
       build_type: pull-request
       matrix_name: conda-cpp-build
   rocky8-clib-standalone-build:
-    needs: [build-details, rocky8-clib-standalone-build-matrix]
+    needs: [rocky8-clib-standalone-build-matrix]
+    runs-on: linux-${{ matrix.ARCH }}-cpu16
+    env:
+      RAPIDS_BUILD_TYPE: pull-request
     permissions:
       actions: read
       contents: read
       id-token: write
       packages: read
       pull-requests: read
-    secrets: inherit # zizmor: ignore[secrets-inherit]
-    uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@release/26.10
     strategy:
       fail-fast: false
       matrix: ${{ fromJSON(needs.rocky8-clib-standalone-build-matrix.outputs.matrix) }}
-    with:
-      build_type: pull-request
-      build-datetime: ${{ needs.build-details.outputs.build-datetime }}
-      arch: "${{matrix.arch}}"
-      date: ${{ inputs.date }}_c
-      container_image: "rapidsai/ci-wheel:26.10-cuda${{ matrix.CUDA_VER }}-${{ matrix.LINUX_VER }}-py${{ matrix.PY_VER }}"
-      node_type: "cpu16"
-      requires_license_builder: true
-      script: "ci/build_standalone_c.sh --build-tests"
-      artifact-name: "libcuvs_c_${{ matrix.CUDA_VER }}_${{ matrix.ARCH }}.tar.gz"
-      file_to_upload: "libcuvs_c.tar.gz"
-      sha: ${{ inputs.sha }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          ref: ${{ inputs.sha || github.sha }}
+          fetch-depth: 0
+          persist-credentials: false
+      - uses: aws-actions/configure-aws-credentials@cbe3b392738ccf3f987d68400dafcf4b0624a56c # v6.2.4
+        with:
+          role-to-assume: ${{ vars.AWS_ROLE_ARN }}
+          aws-region: ${{ vars.AWS_REGION }}
+          role-duration-seconds: 10800
+      - name: Setup sccache-dist
+        uses: rapidsai/shared-actions/setup-sccache-dist@main
+        with:
+          request-timeout: 7140
+      - name: Build standalone C tarball
+        env:
+          CUVS_TARBALL_CUDA_VERSION: ${{ matrix.CUDA_VER }}
+          CUVS_TARBALL_PYTHON_VERSION: ${{ matrix.PY_VER }}
+        run: |
+          ./build.sh tarball --tarball-build-tests
+      - name: Upload artifacts
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: libcuvs_c_${{ matrix.CUDA_VER }}_${{ matrix.ARCH }}.tar.gz
+          path: libcuvs_c.tar.gz
   rocky8-clib-tests-matrix:
     needs: [rocky8-clib-standalone-build, changed-files]
     permissions:

.gitignore

@@ -19,6 +19,7 @@ dist/
 python/**/**/*.cpp
 python/cuvs/record.txt
 log
+infrequent_licenses/
 .ipynb_checkpoints
 .DS_Store
 dask-worker-space/
@@ -29,6 +30,7 @@ temporary_*.json
 rust/target/
 rust/Cargo.lock
 rmm_log.txt
+*.tar.gz
 
 ## example notebooks
 notebooks/simplewiki-2020-11-01-nq-distilbert-base-v1.pt

Dockerfile.standalone

@@ -0,0 +1,23 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Environment and runner for the standalone C build.
+#
+# End-user instructions: https://docs.nvidia.com/cuvs/installation#build-the-standalone-c-library-with-docker
+
+ARG CUDA_VERSION=notset
+ARG PYTHON_VERSION=notset
+ARG RAPIDS_VERSION=notset
+FROM rapidsai/ci-wheel:${RAPIDS_VERSION}-cuda${CUDA_VERSION}-rockylinux8-py${PYTHON_VERSION}
+
+# Output directory for the standalone archive. Bind-mount a host folder here
+# (e.g. -v $(pwd)/build:/build) so libcuvs_c.tar.gz is written to the host.
+ENV CUVS_TARBALL_BUILD_OUTPUT_DIR=/build
+ENV CUVS_TARBALL_IN_CONTAINER=1
+
+# Run from repo root; the repo is expected to be bind-mounted at /workspace.
+WORKDIR /workspace
+
+# Run the build script, which writes the standalone tarball to the mounted output dir.
+# Example: docker run -v $(pwd):/workspace -v $(pwd)/build:/build cuvs-standalone-c [--tarball-build-tests]
+ENTRYPOINT ["/bin/bash", "-c", "ci/standalone_c/build_standalone_c.sh \"$@\"", "--"]

README.md

@@ -84,6 +84,10 @@ cuVS comes with pre-built packages that can be installed through [conda](https:/
 
 Please see the [Build and Install Guide](https://docs.rapids.ai/api/cuvs/nightly/build/) for more information on installing the available cuVS packages and building from source.
 
+### Standalone C library (Docker)
+
+To build the standalone C library tarball (`libcuvs_c.tar.gz`) for use in your own C/C++ projects, see **[Building the standalone C library with Docker](https://docs.nvidia.com/cuvs/installation#build-the-standalone-c-library-with-docker)**.
+
 ## Getting Started
 
 The following code snippets train an approximate nearest neighbors index for the CAGRA algorithm in the various different languages supported by cuVS.

build.sh

@@ -19,7 +19,7 @@ ARGS=$*
 # scripts, and that this script resides in the repo dir!
 REPODIR=$(cd "$(dirname "$0")"; pwd)
 
-VALIDARGS="clean libcuvs python rust go java lucene docs tests bench-ann examples --uninstall  -v -g -n --allgpuarch --no-mg --mnmg-tests --no-cpu --cpu-only --no-shared-libs --no-nvtx --show_depr_warn --incl-cache-stats --time -h --run-java-tests --build-java-examples"
+VALIDARGS="clean libcuvs python rust go java lucene docs tests bench-ann examples tarball --tarball-build-tests --uninstall  -v -g -n --allgpuarch --no-mg --mnmg-tests --no-cpu --cpu-only --no-shared-libs --no-nvtx --show_depr_warn --incl-cache-stats --time -h --run-java-tests --build-java-examples"
 HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<tool>] [--limit-tests=<targets>] [--limit-bench-ann=<targets>] [--build-metrics=<filename>]
  where <target> is:
    clean            - remove all existing build artifacts and configuration (start over)
@@ -34,6 +34,7 @@ HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<to
    tests            - build the tests
    bench-ann        - build end-to-end ann benchmarks
    examples         - build the examples
+   tarball          - build the standalone C library tarball with Docker
 
  and <flag> is:
    -v                          - verbose build mode
@@ -59,6 +60,7 @@ HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<to
    --run-java-tests            - run Java tests after building
    --build-java-examples       - also build the examples/java/cuvs-java and examples/java/cuvs-lucene
                                  projects against the jars just built (with the 'java'/'lucene' targets)
+   --tarball-build-tests       - include the C library tests in the standalone tarball
    --build-metrics             - filename for generating build metrics report for libcuvs
    --incl-cache-stats          - include cache statistics in build metrics report
    --cmake-args=\\\"<args>\\\" - pass arbitrary list of CMake configuration options (escape all quotes in argument)
@@ -607,3 +609,56 @@ if hasArg examples; then
     ./build.sh
     popd
 fi
+
+################################################################################
+# Build the standalone C library tarball (if requested)
+
+if hasArg tarball; then
+    if [[ "${CUVS_TARBALL_IN_CONTAINER:-0}" == "1" ]]; then
+        CUVS_TARBALL_BUILD_OUTPUT_DIR="${CUVS_TARBALL_BUILD_OUTPUT_DIR:-${REPODIR}}"
+        tar czf "${CUVS_TARBALL_BUILD_OUTPUT_DIR}/libcuvs_c.tar.gz" -C "${REPODIR}/c/build/install" .
+    else
+        CUVS_TARBALL_CUDA_VERSION="${CUVS_TARBALL_CUDA_VERSION:-13.3.0}"
+        CUVS_TARBALL_PYTHON_VERSION="${CUVS_TARBALL_PYTHON_VERSION:-3.14}"
+        CUVS_TARBALL_BUILD_OUTPUT_DIR="${CUVS_TARBALL_BUILD_OUTPUT_DIR:-${REPODIR}/build}"
+        CUVS_TARBALL_IMAGE_NAME="nvidia/cuvs-standalone-c:local-cuda${CUVS_TARBALL_CUDA_VERSION}-py${CUVS_TARBALL_PYTHON_VERSION}"
+
+        mkdir -p "${CUVS_TARBALL_BUILD_OUTPUT_DIR}"
+        BUILD_OUTPUT_DIR_ABS=$(realpath "${CUVS_TARBALL_BUILD_OUTPUT_DIR}")
+
+        echo "Building Docker image ${CUVS_TARBALL_IMAGE_NAME} (CUDA ${CUVS_TARBALL_CUDA_VERSION}, Python ${CUVS_TARBALL_PYTHON_VERSION})..."
+        docker build \
+            -f "${REPODIR}/Dockerfile.standalone" \
+            --build-arg CUDA_VERSION="${CUVS_TARBALL_CUDA_VERSION}" \
+            --build-arg PYTHON_VERSION="${CUVS_TARBALL_PYTHON_VERSION}" \
+            --build-arg RAPIDS_VERSION="${RAPIDS_VERSION_MAJOR_MINOR}" \
+            -t "${CUVS_TARBALL_IMAGE_NAME}" \
+            "${REPODIR}"
+
+        # optionally pass additional arguments through to the container's entrypoint
+        DOCKER_ENTRYPOINT_ARGS=()
+        if hasArg --tarball-build-tests; then
+            DOCKER_ENTRYPOINT_ARGS+=(--tarball-build-tests)
+        fi
+
+        echo "Running standalone C build in container..."
+        # NOTE: the '--env-file' trick for AWS credentials keeps them out of 'ps aux' / 'docker ps' output
+        docker run \
+            --rm \
+            -v "${REPODIR}:/workspace:rw" \
+            -v "${BUILD_OUTPUT_DIR_ABS}:/build:rw" \
+            --env CI="${CI:-false}" \
+            --env PARALLEL_LEVEL="${PARALLEL_LEVEL}" \
+            --env RAPIDS_BUILD_TYPE="${RAPIDS_BUILD_TYPE:-}" \
+            --env-file <(env | grep -E '^AWS_(ACCESS_KEY_ID|SECRET_ACCESS_KEY|SESSION_TOKEN)=') \
+            --env-file <(env | grep -E '^SCCACHE_.*=') \
+            "${CUVS_TARBALL_IMAGE_NAME}" \
+            "${DOCKER_ENTRYPOINT_ARGS[@]}"
+
+        cp -v "${BUILD_OUTPUT_DIR_ABS}/libcuvs_c.tar.gz" "${REPODIR}/libcuvs_c.tar.gz"
+        echo "Copied libcuvs_c.tar.gz to ${REPODIR}/libcuvs_c.tar.gz"
+    fi
+
+    # print contents of the tarball
+    ls -lh "${CUVS_TARBALL_BUILD_OUTPUT_DIR}/libcuvs_c.tar.gz"
+fi

ci/build_standalone_c.sh ci/standalone_c/build_standalone_c.shci/build_standalone_c.sh renamed to ci/standalone_c/build_standalone_c.sh

@@ -1,67 +1,47 @@
 #!/bin/bash
 # SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
+#
+# Build script for the standalone C library.
+#
+# Use 'Dockerfile.standalone' to build an image with all the prerequisites
+# and run this in a container.
 
 set -euo pipefail
 
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+
 TOOLSET_VERSION=14
-NINJA_VERSION=v1.13.1
 
 BUILD_C_LIB_TESTS="OFF"
-if [[ "${1:-}" == "--build-tests" ]]; then
+if [[ "${1:-}" == "--tarball-build-tests" ]]; then
   BUILD_C_LIB_TESTS="ON"
 fi
 
-dnf install -y \
-      patch \
-      tar \
-      unzip \
-      wget
-
-if ! command -V ninja >/dev/null 2>&1; then
-    case "$(uname -m)" in
-        x86_64)
-            wget --no-hsts -q -O /tmp/ninja-linux.zip "https://github.com/ninja-build/ninja/releases/download/${NINJA_VERSION}/ninja-linux.zip";
-            ;;
-        aarch64)
-            wget --no-hsts -q -O /tmp/ninja-linux.zip "https://github.com/ninja-build/ninja/releases/download/${NINJA_VERSION}/ninja-linux-aarch64.zip";
-            ;;
-        *)
-            echo "Unrecognized platform '$(uname -m)'" >&2
-            exit 1
-            ;;
-    esac
-    unzip -d /usr/bin /tmp/ninja-linux.zip
-    chmod +x /usr/bin/ninja
-    rm /tmp/ninja-linux.zip
-fi
-
 source rapids-install-sccache
 source rapids-configure-sccache
-source rapids-datetime-string
 
-rapids-pip-retry install cmake
+PIP_PACKAGES=(
+  'cmake>=4.0'
+  'git+https://github.com/rapidsai/spdx-license-builder.git'
+  'ninja>=1.13'
+)
 
 RAPIDS_CUDA_MAJOR="${RAPIDS_CUDA_VERSION%%.*}"
 if [[ "${RAPIDS_CUDA_MAJOR}" == "13" ]]; then
-  rapids-pip-retry install cuda-tile "cuda-toolkit[tileiras]==${RAPIDS_CUDA_VERSION%.*}.*"
+  PIP_PACKAGES+=(
+    cuda-tile
+  )
 fi
 
+rapids-pip-retry install "${PIP_PACKAGES[@]}"
 pyenv rehash
 
-rapids-print-env
-
 rapids-logger "Begin cpp build"
 
 sccache --stop-server 2>/dev/null || true
 
-RAPIDS_PACKAGE_VERSION=$(rapids-generate-version)
-export RAPIDS_PACKAGE_VERSION
-
-RAPIDS_ARTIFACTS_DIR=${RAPIDS_ARTIFACTS_DIR:-"${PWD}/artifacts"}
-mkdir -p "${RAPIDS_ARTIFACTS_DIR}"
-export RAPIDS_ARTIFACTS_DIR
-
 scl enable gcc-toolset-${TOOLSET_VERSION} -- \
       cmake -S cpp -B cpp/build/ -GNinja \
             -DCMAKE_CUDA_HOST_COMPILER=/opt/rh/gcc-toolset-${TOOLSET_VERSION}/root/usr/bin/gcc \
@@ -107,11 +87,8 @@ if [ "${BUILD_C_LIB_TESTS}" != "OFF" ]; then
       cmake --install c/build --prefix c/build/install --component testing
 fi
 
-
 rapids-logger "Begin gathering licenses"
-rapids-pip-retry install git+https://github.com/rapidsai/spdx-license-builder.git
 license-builder . --output-json c/build/install/licenses.json --output-txt c/build/install/LICENSE
 
 rapids-logger "Begin c tarball creation"
-tar czf libcuvs_c.tar.gz -C c/build/install/ .
-ls -lh libcuvs_c.tar.gz
+"${REPO_ROOT}/build.sh" tarball