Publish dev versions of the ESP-IDF component from the nightly branch

Projects that need the component before the next release can now pin a
dev version from the registry instead of generating the component from
an ExecuTorch checkout. A manual run of publish-espressif-idf.yml on the
nightly branch with version_suffix -dev publishes X.Y.Z-dev.YYYYMMDD of
that branch's commit, through the same packing, testing, review and
registry checks as a release. The generator's new --version-suffix -dev
stamps HEAD's commit date, and like the other published stamps checks
the checkout: HEAD must be on the nightly branch, checked out or fetched
from origin, and free of uncommitted changes to what the generator
reads. --allow-dev-stamp stays the unchecked stamp for local builds.

A dev version's kernel bundles need executorch from the component's
exact commit. ExecuTorch's nightly wheels are built from the nightly
branch's commits, so the plan refuses -dev on any other ref and any
other suffix on the nightly branch. That check runs from the nightly
branch's copy of the workflow, which is main's, so it guards against
mistakes rather than changes to it: the runbook protects the nightly
branch with a ruleset before the environment trusts it, and has the
approver of a dev upload review what changed since the last approved one
in every file the environment's jobs run: the workflow, the upload
script and its requirements, and the test script whose fetch mode they
run. Before the build, the new wheel mode of
test_espressif_idf_nightly.sh checks that the nightly index has the
version's wheels for every platform and CPython the wheel workflows
build, and that all of them record that commit, which takes about 30
seconds. It refuses a version dated today, since a second nightly the
same day, which has happened, replaces that day's wheels; the runbook
has the workflow run between 00:00 UTC and the next nightly, and once
with dry_run set before the first dev upload. The nightly matrix gains a
job, which its pull requests run, that checks the wheel check's refusals
and runs it on the finished nightlies of the last 60 days that have
every wheel, newest first, until one passes, since a second nightly the
same day can leave a day's wheels from two commits. It fails if none
passes, or if the index has every wheel of a day no nightly commit is
dated, which means the dev version's dates no longer match the wheels';
otherwise nightly wheels can fail to build for days, so finding none
only warns.

ExecuTorch's wheels declare no torch, so the component's VERSION now
records the torch requirement install_requirements.py builds the wheels
against, and the CPU index of its TORCH_URL_BASE, which it installs
torch from and which also serves torch release candidates. The generator
reads both before --version exits, so the stamp checks catch a pin it
cannot read, and counts install_requirements.py among its inputs. The
new torch mode checks that the requirement resolves from that index for
every platform and CPython, in the package job before upload and in the
nightly's prepare job every day, and the component test checks both
lines. The component's README gives the exact pip commands for its
version, a release's, an RC's or a dev version's, and the component test
checks the release's and the RC's. torch comes from its own index in a
command of its own, because download.pytorch.org's test and release
indexes also serve executorch builds, as X.Y.Z+cpu, which pip would
prefer to PyPI's X.Y.Z and which gen_kernels.py accepts for a release
without a commit check. and gen_kernels.py names the nightly wheel a dev
version needs. The nightly index keeps about 60 days of wheels, so a
project moves to a newer dev version, or to the release, before its pin
ages out; firmware builds from a committed kernel bundle are unaffected.

Dev versions are semver pre-releases, which a version range picks up
only if it opts in, so they go to production without reaching projects
on releases. They read 1.6.0-dev.YYYYMMDD, as the nightly wheels do,
until main's version.txt moves past 1.6.0, and sort below 1.6.0
meanwhile; exact pins are unaffected. Dev runs queue in the nightly
branch's concurrency group, so one cannot replace a pending release run.

Before the first dev upload, a repo admin protects the nightly branch
and adds it to the deployment branches of the esp-component-registry
environment. The workflow accepts -dev once this change reaches the
nightly branch.

Authored with Claude Code.

Author: rascani

.ci/scripts/test_espressif_idf_component.sh

@@ -92,19 +92,36 @@ git -C "${release_clone}" tag -f "v${release_version}" > /dev/null
 bash "${release_clone}/examples/espressif/build_idf_component.sh" --version-suffix "" \
     --out "${build_dir}/release_component" 2>&1 | tee "${build_dir}/generate-release.log"
 compote manifest lint "${build_dir}/release_component"
-python - "${build_dir}/release_component" "${release_version}" "$(git rev-parse HEAD)" <<'PY'
+python - "${build_dir}/release_component" "${release_version}" "$(git rev-parse HEAD)" \
+    "${release_clone}" <<'PY'
+import re
 import sys
 from pathlib import Path
 
-component, version, commit = Path(sys.argv[1]), sys.argv[2], sys.argv[3]
+component, version, commit, clone = Path(sys.argv[1]), sys.argv[2], sys.argv[3], sys.argv[4]
 lines = (component / "VERSION").read_text().splitlines()
 for line in (f"component_version: {version}", f"executorch_commit: {commit}"):
     if line not in lines:
         sys.exit(f"FAIL: the release component's VERSION has no '{line}'")
 if not any(l.startswith("generator: ") and l.endswith(" --version-suffix ''") for l in lines):
     sys.exit("FAIL: the release component's VERSION does not record --version-suffix ''")
+sys.path.insert(0, clone)
+from install_requirements import TORCH_URL_BASE
+
+if not any(re.fullmatch(r"torch_requirement: torch==\d+\.\d+\.\d+\S*", l) for l in lines):
+    sys.exit("FAIL: the release component's VERSION has no torch_requirement: torch==X.Y.Z")
+if f"torch_index: {TORCH_URL_BASE}/cpu" not in lines:
+    sys.exit(f"FAIL: the release component's VERSION has no torch_index: {TORCH_URL_BASE}/cpu")
 if f'version: "{version}"' not in (component / "idf_component.yml").read_text().splitlines():
     sys.exit(f"FAIL: the release component's idf_component.yml is not version {version}")
+readme = (component / "README.md").read_text().splitlines()
+torch = next(l.split(": ", 1)[1] for l in lines if l.startswith("torch_requirement: "))
+for line in (
+    f'pip install "{torch}" --index-url {TORCH_URL_BASE}/cpu',
+    f'pip install "executorch=={version}"',
+):
+    if line not in readme:
+        sys.exit(f"FAIL: the release component's README.md has no '{line}'")
 print(f"OK: a clean checkout tagged v{version} generates release {version}")
 PY
 # An RC at the same commit, which a release commit can also be.
@@ -116,6 +133,11 @@ if ! grep -qxF "component_version: ${release_version}-rc.1" "${build_dir}/rc_com
     echo "FAIL: the RC component's VERSION is not ${release_version}-rc.1" >&2
     exit 1
 fi
+rc_install="pip install \"executorch==${release_version}\" --extra-index-url https://download.pytorch.org/whl/test/cpu"
+if ! grep -qxF "${rc_install}" "${build_dir}/rc_component/README.md"; then
+    echo "FAIL: the RC component's README.md has no '${rc_install}'" >&2
+    exit 1
+fi
 echo "OK: the clean checkout tagged v${release_version}-rc1 generates RC ${release_version}-rc.1"
 gen_kernels() {
     local out=$1

.ci/scripts/test_espressif_idf_nightly.sh

@@ -34,17 +34,39 @@
 #     pytorch/executorch, builds the smoke consumer for ESP32-S3 with it, fails
 #     if the registry serves files other than <dir>'s archive, and runs the
 #     build on the emulator.
+#   test_espressif_idf_nightly.sh wheel <dir> <dev version> <commit>
+#     Checks that the ExecuTorch nightly wheels of the version's date exist for
+#     every platform and CPython and were built from <commit>, so a project can
+#     install the executorch its kernel bundles need. Refuses today's nightly,
+#     which a later one could replace. Logs go under <dir>.
+#   test_espressif_idf_nightly.sh rehearse <dir>
+#     Checks that wheel refuses a version that is not a dev version and today's
+#     nightly, then runs it on the finished nightlies of origin/nightly's last
+#     60 days, the nightly index's window, that have every wheel, newest first,
+#     until one passes. Nightly wheels can fail to build for days, so finding
+#     none only warns, unless the index has every wheel of a day no nightly
+#     commit is dated, which means the dev version's date no longer matches
+#     the wheels'.
+#   test_espressif_idf_nightly.sh torch <dir>
+#     Needs only <dir> from prepare: checks that the torch the packed
+#     component's VERSION names resolves from its index for every platform and
+#     CPython.
 
 set -euo pipefail
 
 et_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
-usage="usage: $0 prepare <dir> [stamp options] | build <dir> <shard> <shards> | fetch <dir> <URL> <sha256> | registry <dir> <URL>"
+usage="usage: $0 prepare <dir> [stamp options] | build <dir> <shard> <shards> | fetch <dir> <URL> <sha256> | registry <dir> <URL> | wheel <dir> <dev version> <commit> | rehearse <dir> | torch <dir>"
 mode=${1:?${usage}}
 dir=${2:?${usage}}
 mkdir -p "${dir}"
 dir=$(cd "${dir}" && pwd)
 cd "${et_root}"
 targets=(esp32 esp32s3 esp32c3 esp32c5 esp32c6 esp32p4)
+# The platforms and CPython versions ExecuTorch's wheels are built for, as
+# .github/workflows/build-wheels-{linux,aarch64-linux,macos,windows}.yml set them.
+wheel_platforms=(manylinux_2_28_x86_64 manylinux_2_28_aarch64 macosx_14_0_arm64 win_amd64)
+wheel_pythons=(3.11 3.12 3.13 3.14 3.15)
+nightly_index=https://download.pytorch.org/whl/nightly/cpu
 
 # Runs a command with its output in a log, and shows the log's tail if it fails.
 logged() {
@@ -400,6 +422,145 @@ settings_applied() {
     [[ "${applied}" == true ]]
 }
 
+# Prints each wheel a dev version lacks: version, the nightly index's listing.
+missing_wheels() {
+    local listed platform python
+    listed=$(grep -F "executorch-${1/-dev./.dev}" <<< "$2" || true)
+    for platform in "${wheel_platforms[@]}"; do
+        for python in "${wheel_pythons[@]}"; do
+            grep -qF -- "-cp${python/./}-cp${python/./}-${platform}.whl" <<< "${listed}" \
+                || echo "cp${python/./}-${platform}"
+        done
+    done
+}
+
+# Checks a dev version's nightly wheels: version, commit.
+check_wheels() {
+    local version=$1 commit=$2 listing missing wheels platform python
+    if [[ "${version}" != *-dev.* ]]; then
+        echo "FAIL: ${version} is not a dev version" >&2
+        exit 1
+    fi
+    # A second nightly the same day would replace the wheels of this one.
+    if [[ "${version##*-dev.}" == "$(date -u +%Y%m%d)" ]]; then
+        echo "FAIL: ${version} is today's nightly, which a later nightly today could replace;" \
+            "publish it after 00:00 UTC, before the next nightly." >&2
+        exit 1
+    fi
+    # The version's date is its nightly commit's UTC date, and the wheels' is
+    # the day they were built; when the two differ, no wheels are found.
+    listing=$(curl -sSf --retry 3 "${nightly_index}/executorch/")
+    missing=$(missing_wheels "${version}" "${listing}")
+    if [[ -n "${missing}" ]]; then
+        echo "FAIL: the nightly index has no ${version} wheel for ${missing//$'\n'/ }" >&2
+        exit 1
+    fi
+    wheels=$(mktemp -d)
+    # shellcheck disable=SC2064 # wheels is local, so expand it now.
+    trap "rm -rf '${wheels}'" EXIT
+    mkdir -p "${dir}/logs"
+    for platform in "${wheel_platforms[@]}"; do
+        for python in "${wheel_pythons[@]}"; do
+            logged "${dir}/logs/nightly-wheel-cp${python/./}-${platform}.log" python -m pip download \
+                --no-deps --only-binary=:all: --platform "${platform}" \
+                --python-version "${python}" --implementation cp --index-url "${nightly_index}" \
+                --dest "${wheels}" "executorch==${version/-dev./.dev}"
+        done
+    done
+    python - "${commit}" "${wheels}"/executorch-*.whl <<'PY'
+import re
+import sys
+import zipfile
+
+commit, wheels = sys.argv[1], sys.argv[2:]
+wrong = []
+for wheel in wheels:
+    version_py = zipfile.ZipFile(wheel).read("executorch/version.py").decode()
+    built = re.search(r"^git_version: .* = '(\w+)'", version_py, re.MULTILINE)
+    if not built or built.group(1) != commit:
+        wrong.append(f"{wheel} ({built.group(1) if built else 'no commit'})")
+if wrong:
+    sys.exit(f"FAIL: not built from the component's {commit}: {', '.join(wrong)}")
+print(f"OK: all {len(wheels)} wheels were built from {commit}")
+PY
+}
+
+wheel() {
+    check_wheels "${3:?${usage}}" "${4:?${usage}}"
+}
+
+rehearse() {
+    local today output listing commit day version
+    today=$(date -u +%Y%m%d)
+    output=$( (check_wheels 1.6.0 0) 2>&1 || true)
+    if [[ "${output}" != *"is not a dev version"* ]]; then
+        echo "FAIL: the wheel check did not refuse 1.6.0: ${output}" >&2
+        exit 1
+    fi
+    output=$( (check_wheels "1.6.0-dev.${today}" 0) 2>&1 || true)
+    if [[ "${output}" != *"is today's nightly"* ]]; then
+        echo "FAIL: the wheel check did not refuse today's nightly: ${output}" >&2
+        exit 1
+    fi
+    echo "OK: the wheel check refuses a version that is not a dev version and today's nightly"
+    listing=$(curl -sSf --retry 3 "${nightly_index}/executorch/")
+    local failures=()
+    for commit in $(git rev-list --first-parent --since=60.days origin/nightly); do
+        day=$(TZ=UTC git log -1 --format=%cd --date=format-local:%Y%m%d "${commit}")
+        version="$(git show "${commit}:version.txt")-dev.${day}"
+        if [[ "${day}" != "${today}" && -z "$(missing_wheels "${version}" "${listing}")" ]]; then
+            # A second nightly the same day replaces the wheels it builds, so
+            # a day's wheels can come from two commits; an older day can pass.
+            if output=$( (check_wheels "${version}" "${commit}") 2>&1); then
+                echo "${output}"
+                return
+            fi
+            failures+=("${version}: ${output##*$'\n'}")
+        fi
+    done
+    if (( ${#failures[@]} )); then
+        printf 'FAIL: %s\n' "${failures[@]}" >&2
+        exit 1
+    fi
+    # A day inside the commits' window, whose edge can differ by a day.
+    local since listed
+    since=$(date -u -d '59 days ago' +%Y%m%d)
+    for listed in $(grep -oE 'executorch-[0-9]+\.[0-9]+\.[0-9]+\.dev[0-9]{8}' <<< "${listing}" \
+        | sort -u); do
+        day=${listed##*.dev}
+        version="${listed#executorch-}"
+        version="${version%.dev*}-dev.${day}"
+        if [[ "${day}" > "${since}" && "${day}" != "${today}" ]] \
+            && [[ -z "$(missing_wheels "${version}" "${listing}")" ]]; then
+            echo "FAIL: the nightly index has every wheel of ${version}, but no nightly commit" \
+                "of that version is dated ${day}; the dev version's date no longer matches the" \
+                "wheels'." >&2
+            exit 1
+        fi
+    done
+    echo "::warning::no nightly of the last 60 days has every wheel; the dev wheel check went unrehearsed."
+}
+
+torch() {
+    local stamp requirement index target platform python
+    stamp=$(tar -xzOf "${dir}/dist/executorch_$(cat "${dir}/version").tgz" ./VERSION)
+    requirement=$(sed -n 's/^torch_requirement: *//p' <<< "${stamp}")
+    index=$(sed -n 's/^torch_index: *//p' <<< "${stamp}")
+    target=$(mktemp -d)
+    # shellcheck disable=SC2064 # target is local, so expand it now.
+    trap "rm -rf '${target}'" EXIT
+    mkdir -p "${dir}/logs"
+    for platform in "${wheel_platforms[@]}"; do
+        for python in "${wheel_pythons[@]}"; do
+            logged "${dir}/logs/torch-cp${python/./}-${platform}.log" python -m pip install --dry-run \
+                --no-deps --only-binary=:all: --platform "${platform}" \
+                --python-version "${python}" --implementation cp --target "${target}" \
+                --index-url "${index}" "${requirement}"
+        done
+    done
+    echo "OK: ${requirement} resolves from ${index} for CPython ${wheel_pythons[*]} on ${wheel_platforms[*]}"
+}
+
 # Runs a build on its emulator: build dir, target, log, then idf.py arguments.
 run_firmware() {
     local build=$1 target=$2 log=$3
@@ -434,5 +595,8 @@ case "${mode}" in
     build) build "$@" ;;
     fetch) fetch "$@" ;;
     registry) registry "$@" ;;
+    wheel) wheel "$@" ;;
+    rehearse) rehearse ;;
+    torch) torch ;;
     *) echo "unknown mode ${mode}; ${usage}" >&2; exit 1 ;;
 esac

.github/workflows/nightly-espressif-idf.yml

@@ -50,13 +50,34 @@ jobs:
         source .ci/scripts/utils.sh
         install_executorch --use-pt-pinned-commit --minimal
         bash .ci/scripts/test_espressif_idf_nightly.sh prepare "${RUNNER_ARTIFACT_DIR}"
+        bash .ci/scripts/test_espressif_idf_nightly.sh torch "${RUNNER_ARTIFACT_DIR}"
         # Runs the release stamp's checks of the checkout, which publishing
         # otherwise first runs at a real tag, at a throwaway local one. Only a
         # publish run reaches the check of the executorch package, which
         # --version stops before.
         git tag -f "v$(cat version.txt)"
         bash examples/espressif/build_idf_component.sh --version --version-suffix ""
 
+  # The dev versions' wheel check, which publishing otherwise first runs on a
+  # dispatch from the nightly branch.
+  dev-wheel-check:
+    name: Rehearse the dev version's wheel check
+    runs-on: ubuntu-22.04
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+        with:
+          ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: '3.11'
+      - run: |
+          set -euo pipefail
+          git fetch -q --shallow-since='60 days ago' origin \
+            +refs/heads/nightly:refs/remotes/origin/nightly
+          bash .ci/scripts/test_espressif_idf_nightly.sh rehearse "${RUNNER_TEMP}/wheel"
+
   registry-dry-run:
     name: Registry validation (upload --dry-run)
     needs: [docker-image, prepare]

.github/workflows/publish-espressif-idf.yml

@@ -1,11 +1,12 @@
 name: Publish ESP-IDF component
 
 # Publishes pytorch/executorch to the ESP Component Registry: RC tags to the
-# staging registry, and release and respin tags to staging and then
-# production; manual runs validate without uploading unless told otherwise.
-# The archive is built and tested on the ESP-IDF releases the component
-# supports before any upload, and each upload is followed by a consumer build
-# that resolves the version from that registry. See the release runbook in
+# staging registry, release and respin tags to staging and then production,
+# and on demand a dev version of a nightly branch commit for early adopters;
+# other manual runs validate without uploading unless told otherwise. The
+# archive is built and tested on the ESP-IDF releases the component supports
+# before any upload, and each upload is followed by a consumer build that
+# resolves the version from that registry. See the release runbook in
 # examples/espressif/README.md.
 
 on:
@@ -15,7 +16,7 @@ on:
   workflow_dispatch:
     inputs:
       version_suffix:
-        description: 'Version suffix: empty at a release tag, -rc.N at an RC tag, ~N at a respin tag vX.Y.Z-esp-idf.N'
+        description: 'Version suffix: empty at a release tag, -rc.N at an RC tag, ~N at a respin tag vX.Y.Z-esp-idf.N, -dev for the dev version of the nightly branch'
         type: string
         default: ''
       registry:
@@ -26,7 +27,7 @@ on:
           - production
         default: staging
       dry_run:
-        description: 'Only validate: pack, test and dry-run the uploads, uploading nothing. Run on the release, RC or respin tag'
+        description: 'Only validate: pack, test and dry-run the uploads, uploading nothing. Run on the release, RC or respin tag, or for -dev the nightly branch'
         type: boolean
         default: true
   pull_request:
@@ -62,6 +63,7 @@ jobs:
         shell: bash
         env:
           EVENT: ${{ github.event_name }}
+          REF: ${{ github.ref }}
           REF_NAME: ${{ github.ref_name }}
           VERSION_SUFFIX: ${{ inputs.version_suffix }}
           REGISTRY: ${{ inputs.registry }}
@@ -87,13 +89,21 @@ jobs:
               fi
               ;;
             workflow_dispatch)
-              # The generator stamps a release, RC or respin only at its tag.
-              if [[ "${GITHUB_REF}" != refs/tags/v* ]]; then
-                echo "::error::run this workflow on the release, RC or respin tag, not ${GITHUB_REF}."
+              if [[ ! "${VERSION_SUFFIX}" =~ ^(-rc\.[1-9][0-9]*|~[1-9][0-9]*|-dev)?$ ]]; then
+                echo "::error::version_suffix must be empty, -rc.N, ~N or -dev, not '${VERSION_SUFFIX}'."
                 exit 1
               fi
-              if [[ ! "${VERSION_SUFFIX}" =~ ^(-rc\.[1-9][0-9]*|~[1-9][0-9]*)?$ ]]; then
-                echo "::error::version_suffix must be empty, -rc.N or ~N, not '${VERSION_SUFFIX}'."
+              # A dev version's users install the nightly wheel of its commit,
+              # and the generator stamps a release, RC or respin only at its
+              # tag. This runs from the dispatched ref's copy of the workflow,
+              # so the environment's reviewers check what that copy does.
+              if [[ "${VERSION_SUFFIX}" == -dev ]]; then
+                if [[ "${REF}" != refs/heads/nightly ]]; then
+                  echo "::error::a dev version is published from the nightly branch, not ${REF}."
+                  exit 1
+                fi
+              elif [[ "${REF}" != refs/tags/v* ]]; then
+                echo "::error::run this workflow on the release, RC or respin tag, not ${REF}."
                 exit 1
               fi
               if [[ "${VERSION_SUFFIX}" == -rc.* && "${REGISTRY}" == production ]]; then
@@ -150,22 +160,34 @@ jobs:
         unset SCCACHE_BUCKET SCCACHE_S3_KEY_PREFIX
         CONDA_ENV=$(conda env list --json | jq -r '.envs | .[-1]')
         conda activate "${CONDA_ENV}"
+        suffix="${{ needs.plan.outputs.suffix }}"
+        # A tag names its version, and a dispatch appends its suffix to
+        # version.txt. A dev version takes the generator's, which adds the
+        # commit's date, so for it the comparison below checks only that
+        # prepare stamped the same.
+        expected="${{ needs.plan.outputs.version }}"
+        expected="${expected:-$(cat version.txt)${suffix}}"
+        # Before the build: a dev version's users install the nightly wheels of
+        # its commit.
+        if [[ "${suffix}" == -dev ]]; then
+          expected=$(bash examples/espressif/build_idf_component.sh --version --version-suffix -dev)
+          bash .ci/scripts/test_espressif_idf_nightly.sh wheel "${RUNNER_ARTIFACT_DIR}" \
+            "${expected}" "$(git rev-parse HEAD)"
+        fi
         source .ci/scripts/utils.sh
         install_executorch --use-pt-pinned-commit --minimal
         if [[ "${{ needs.plan.outputs.rehearsal }}" == true ]]; then
           # A pull request rehearses the release stamp at a throwaway local tag.
           git tag -f "v$(cat version.txt)"
         fi
-        # A tag names its version; a dispatch appends its suffix to version.txt.
-        expected="${{ needs.plan.outputs.version }}"
-        expected="${expected:-$(cat version.txt)${{ needs.plan.outputs.suffix }}}"
         bash .ci/scripts/test_espressif_idf_nightly.sh prepare "${RUNNER_ARTIFACT_DIR}" \
-          --version-suffix "${{ needs.plan.outputs.suffix }}"
+          --version-suffix "${suffix}"
         version=$(cat "${RUNNER_ARTIFACT_DIR}/version")
         if [[ "${version}" != "${expected}" ]]; then
           echo "FAIL: generated version ${version}, expected ${expected}" >&2
           exit 1
         fi
+        bash .ci/scripts/test_espressif_idf_nightly.sh torch "${RUNNER_ARTIFACT_DIR}"
         echo "Packed pytorch/executorch ${version}"
         # Every later job reads the tested files from one tar, which
         # keep-archive keeps longer than this artifact.