Portable standalone tarball build with Docker by cjnolet · Pull Request #2443 · NVIDIA/cuvs

cjnolet · 2026-08-13T15:42:49Z

This PR is based on @msarahan's original POC, with the logic of the ci sript moved into build.sh and invoked through the CI script. The readme is also moved into the build and install guide in the docs.

copy-pr-bot · 2026-08-13T15:42:54Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

cjnolet · 2026-08-13T17:28:48Z

/ok to test 0743a95

cjnolet · 2026-08-13T17:55:15Z

/ok to test 05fbbfe

cjnolet · 2026-08-13T18:10:59Z

/ok to test 472089c

cjnolet · 2026-08-13T18:21:50Z

/ok to test 266bd65

cjnolet · 2026-08-13T18:42:07Z

/ok to test 04c2e2b

cjnolet · 2026-08-13T19:27:37Z

/ok to test a0e7f83

jameslamb · 2026-08-14T21:08:50Z

Oh one other thing... after running locally, these files are untracked:

$ git status
infrequent_licenses/
libcuvs_c.tar.gz

I think it'd be worthwhile to add .gitignore rules for them. There aren't any .tar.gz files in source control right now, so I'd support a generic rule like *.tar.gz.

Co-authored-by: James Lamb <jaylamb20@gmail.com>

cjnolet · 2026-08-14T21:39:48Z

/ok to test 1ba4f83

cjnolet · 2026-08-14T23:32:53Z

/ok to test 1ba4f83

cjnolet · 2026-08-14T23:34:41Z

/ok to testr f812e92

cjnolet · 2026-08-15T18:06:14Z

/ok to test f812e92

cjnolet · 2026-08-15T21:28:26Z

/ok to test d073064

copy-pr-bot · 2026-08-15T21:28:30Z

/ok to test d073064

@cjnolet, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

cjnolet · 2026-08-15T21:51:03Z

/ok to test de7e247

cjnolet · 2026-08-18T16:47:34Z

-pyenv rehash
+# rapids-configure-sccache enables anonymous S3 access by default. CI forwards
+# temporary AWS credentials, and sccache rejects both modes at once.
+if [[ -n "${AWS_ACCESS_KEY_ID:-}" ]]; then


@jameslamb I admit I don't know enough about this SCCACHE_S3_NO_CREDENTIALS parameter, but I did get a failure in CI and asked codex to fix it (this is what it did).

cjnolet · 2026-08-18T16:49:01Z

+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
+
+EXTRA_TARBALL_DOCKER_ARGS="--env AWS_ACCESS_KEY_ID \


This seems to be the minimal set of additional args that need to be passed into docker. Please let me know if you disagree @jameslamb. I used this EXTRA_TARBALL_DOCKER_ARGS as the decoupling point.

cjnolet · 2026-08-21T17:46:50Z

/ok to test 86b45f7

jameslamb · 2026-09-11T03:46:52Z

I know you're looking to get this onto release/26.10. It's fallendso far behind and picked up so many merge conflicts that I wasn't confident in updating it and force-pushing.

So I've created a replacement PR targeting release/26.10 over here: #2595

Like this:

git checkout release/26.10
git checkout -b standalone-tarball-build
git merge --squash codex/pr-1814-standalone-build

I'll continue working over there, and then we can come together on how we want it to work and which PR we want to use.

@cjnolet

Replaces #2443 Containerizes the process of building the standalone C library tarballs The starting point for this PR is @cjnolet 's work in #2443, where he wrote this description: > _This PR is based on @msarahan's original POC, with the logic of the ci sript moved into build.sh and invoked through the CI script. The readme is also moved into the build and install guide in the docs._ Changes relative to that PR: * threads `PARALLEL_LEVEL` throught so builds are parallelized * enables the build cluster (`sccache-dist`) in CI * simplifies configuration flow (e.g. reduced duplication of default values, removal of unnecessary flexibility) * makes flow of AWS creds for `sccache` more secure * removes unnecessary configuration * removes unnecessary package installs, consolidates package installs * removes an unnecessary `git clone` of https://github.com/rapidsai/spdx-license-builder * enables `sccache` in CI and with a pattern that could work locally (will share details privately) * updates GitHub Actions third-party actions to their latest versions ## Notes for Reviewers ### How I tested this Locally tried each of the new commands added to `build.md`, with `sccache` enabled. Tried just `build.sh` without `sccache` enabled (that takes a lot longer to run). <details><summary>code for flipping between those (click me)</summary> ```shell # enabling sccache export CI=true # (private steps setting up 'sccache' profile) AWS_ACCESS_KEY_ID=$( aws configure get aws_access_key_id \ --profile sccache ) AWS_SECRET_ACCESS_KEY=$( aws configure get aws_secret_access_key \ --profile sccache ) AWS_SESSION_TOKEN=$( aws configure get aws_session_token \ --profile sccache ) export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN # testing without 'sccache' unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN CI ``` </details> ```shell clean() { sudo rm -rf ./{build,c/build/,cpp/build,libcuvs_c.tar.gz} } # all defaults clean ./build.sh tarball # customizing base image clean CUVS_TARBALL_CUDA_VERSION=12.9.2 \ CUVS_TARBALL_PYTHON_VERSION=3.11 \ ./build.sh tarball # customizing output directory, building tests clean CUVS_TARBALL_BUILD_OUTPUT_DIR="${PWD}/dist" ./build.sh tarball --tarball-build-tests # manual run (no build.sh) clean docker build \ -f Dockerfile.standalone \ --build-arg CUDA_VERSION="13.3.0" \ --build-arg PYTHON_VERSION="3.14" \ --build-arg RAPIDS_VERSION="$(head -1 ./VERSION | cut -d. -f1,2 )" \ -t cuvs-standalone-c:local \ . mkdir -p "${PWD}/dist" docker run --rm \ -v "${PWD}:/workspace" \ -v "${PWD}/dist:/build" \ cuvs-standalone-c:local --tarball-build-tests ``` Saw high cache hit rates from `sccache` and everything working as expected. Also looked at CI logs and saw that fully-cached jobs take around 30 minutes, pretty similar to the timings in CI today. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Mike Sarahan (https://github.com/msarahan) - Corey J. Nolet (https://github.com/cjnolet) URL: #2595

cjnolet requested review from a team as code owners August 13, 2026 15:42

cjnolet requested a review from KyleFromNVIDIA August 13, 2026 15:42

cjnolet self-assigned this Aug 13, 2026

cjnolet added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Aug 13, 2026

cjnolet added this to Unstructured Data Processing Aug 13, 2026

cjnolet moved this to In Progress in Unstructured Data Processing Aug 13, 2026

msarahan and others added 5 commits August 13, 2026 11:53

PoC for Dockerfile for building standalone

0342601

add forgotten standalone files

a936a82

Move standalone tarball build to root

6e1d8b6

Move standalone build instructions to Fern docs

e86938b

Reconcile standalone build with upstream main

3b6132c

cjnolet force-pushed the codex/pr-1814-standalone-build branch from b61beb0 to 3b6132c Compare August 13, 2026 15:55

cjnolet requested a review from a team as a code owner August 13, 2026 15:55

cjnolet added 3 commits August 13, 2026 12:43

Move standalone Docker build into tarball target

f06426b

Rename standalone tarball test flag

eac4049

Merge branch 'main' into codex/pr-1814-standalone-build

0743a95

Normalize standalone Dockerfile copyright

05fbbfe

Fix checkout version annotations in workflows

472089c

Restore executable bit on standalone build script

266bd65

Update devcontainer image references to 26.10

04c2e2b

Fix authenticated sccache configuration

a0e7f83

cjnolet and others added 11 commits August 14, 2026 17:31

Fix standalone Docker build examples

8a7b7fc

Apply suggestion from @jameslamb

b45f475

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

fb6d382

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

c8e26d4

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

2443799

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

bbbe993

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

5419933

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

3ca9566

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

279e6cd

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

143a10e

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Apply suggestion from @jameslamb

1ba4f83

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Fix upload-artifact version annotation

f812e92

cjnolet added 3 commits August 15, 2026 16:12

Refine standalone tarball build setup

6f1cf27

Refine standalone tarball configuration

d073064

Merge branch 'main' into codex/pr-1814-standalone-build

de7e247

Merge branch 'main' into codex/pr-1814-standalone-build

86b45f7

cjnolet commented Aug 18, 2026

View reviewed changes

jameslamb mentioned this pull request Sep 11, 2026

Containerize standalone tarball build #2595

Merged

Uh oh!

Conversation

cjnolet commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 13, 2026

Uh oh!

cjnolet commented Aug 13, 2026

Uh oh!

jameslamb commented Aug 14, 2026

Uh oh!

cjnolet commented Aug 14, 2026

Uh oh!

cjnolet commented Aug 14, 2026

Uh oh!

cjnolet commented Aug 14, 2026

Uh oh!

cjnolet commented Aug 15, 2026

Uh oh!

cjnolet commented Aug 15, 2026

Uh oh!

copy-pr-bot Bot commented Aug 15, 2026

Uh oh!

cjnolet commented Aug 15, 2026

Uh oh!

cjnolet Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

cjnolet Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

cjnolet commented Aug 21, 2026

Uh oh!

jameslamb commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cjnolet commented Aug 13, 2026 •

edited

Loading