Publish the ESP-IDF component to the registry

publish-espressif-idf.yml publishes pytorch/executorch to the ESP
Component Registry. An RC tag vX.Y.Z-rcN publishes X.Y.Z-rc.N to the
staging registry, a release tag vX.Y.Z publishes X.Y.Z to staging and
then production, and a respin tag vX.Y.Z-esp-idf.N, which the generator
requires for X.Y.Z~N since git refs cannot hold ~, publishes that respin
the same way. Other v* tags publish nothing.

The workflow packs the component once, under the generator's
release-stamp checks and with the compiler cache off, so the flatc that
generates the archive's headers and the executorch that exports its
example's model are built fresh, and asserts the version matches the
tag. Before anything is uploaded, it builds and runs that archive in
every consumer configuration on ESP-IDF 6.0.1 and 6.1 and validates it
with both registries. Each upload waits for a reviewer of the
esp-component-registry environment. After each upload, the new registry
mode of test_espressif_idf_nightly.sh polls until the registry lists the
version, builds the consumer with pytorch/executorch resolved from that
registry, checks that the registry served the tested archive's files,
comparing every idf_component.yml as YAML, since the registry rewrites
the component's and each example's with sorted keys and no comments, and
without the examples' override_path, which it drops, and runs it on the
emulator. Production waits for that check on staging. The nightly's
build mode runs the same file check against the mirror, so it is
exercised before the first upload. The consumer takes the registry for
pytorch/executorch as -DET_COMPONENT_REGISTRY_URL.

The jobs after an approval cannot read the tested archive from the run's
GitHub artifact, which lasts 2 days, less than the approvals can take.
The package job puts the tested files in one tar, which every later job
reads, a job records the tar's sha256 as an output before any approval,
and another copies the tar to the run's S3 prefix in gha-artifacts,
where it lasts 14 days, under a name with that hash. The upload and
registry-check jobs fetch it over HTTPS and refuse it unless the hash
matches, since other jobs can write to that bucket. The build and
registry-check jobs keep their logs there too, named per job and
attempt, because an upload to a name already taken fails.

Uploads go through .ci/scripts/upload_espressif_idf_component.sh, which
the nightly's dry-run job now shares. It installs the component manager
from a hash-locked requirements file, wheels only, and the two upload
jobs pin their actions by commit, since anything they run could mint a
registry token. It authenticates with the job's GitHub OIDC token for
the registry, the way espressif/upload-components-ci-action does, and
uploads the archive that was tested, with --allow-existing so a failed
run can be rerun. A version already on the registry is kept, so a rerun
that packed different bytes fails the registry check instead of shipping
them. Since compote then skips the upload and its validation silently,
the script says when the registry already has the version. prepare now
records the commit and generator in that archive at pack time, because
compote applies them only when it packs. Manual runs, which must be on a
release, RC or respin tag since the generator stamps those only at their
tag, validate without uploading unless dry_run is cleared, and every run
that uploads nothing runs the staging upload job with --dry-run outside
the environment, so its runner, actions and archive fetch work before a
release needs them. Pull requests that touch the workflow run it that
way, stamping a release version at a throwaway local tag, because
workflow_dispatch cannot run before the workflow is on main, and the
nightly runs the same release-stamp checks every day, so a change that
would break them shows up before a real tag. Runs share a concurrency
group per tag or branch, so a run waiting for an approval never cancels
another tag's.

The manifest pins esp-nn to the production registry, because the staging
registry records a dependency without one as its own, and staging has no
esp-nn; the component manager treats that as the same source as an
unnamed dependency on the default registry, and the component's README
says where esp-nn comes from.

The README's runbook covers which rerun works when, release branches cut
before the workflow landed, the concurrency group, and the one-time
setup, in order: rulesets that leave v* tags, which the environment
trusts, and release/* branches, which respin tags must be on, to release
managers, then the protected environment, which deploys only from v*
tags, since GitHub creates an unprotected one as soon as a workflow
names it, then the staging namespace, which that registry grants
separately, then each registry's trusted uploader, which names
pytorch/executorch as the repository, publish-espressif-idf.yml as the
workflow and esp-component-registry as the environment, with no branch,
because tag pushes carry none. Reviewers check that a run's ref and
commit are the release's, or for a respin that its tag's commit is on
release/X.Y with reviewed changes; they gate the staging upload on the
nightly's macOS and Windows configure jobs, with ESP-IDF master
advisory, and approve the production upload only once executorch X.Y.Z
is on PyPI, which projects need to make kernel bundles for it. The first
release follows an RC that this workflow published, so the upload and
registry checks have run before a release depends on them.

Authored with Claude Code.

Author: rascani

.ci/scripts/esp_idf_mirror.py

@@ -68,7 +68,7 @@ def main() -> None:
             {
                 "spec": requirement.version_spec,
                 "source": requirement.source.type,
-                "registry_url": None,
+                "registry_url": requirement.registry_url,
                 "is_public": requirement.is_public,
                 "require": requirement.is_required,
                 "name": None if is_idf else name,

.ci/scripts/espressif_idf_upload_requirements.txt

@@ -5,14 +5,15 @@
 # This source code is licensed under the BSD-style license found in the
 # LICENSE file in the root directory of this source tree.
 
-# The ExecuTorch ESP-IDF component's nightly matrix.
+# The ExecuTorch ESP-IDF component's nightly matrix and release checks.
 #
-#   test_espressif_idf_nightly.sh prepare <dir>
+#   test_espressif_idf_nightly.sh prepare <dir> [stamp options]
 #     In the ExecuTorch environment: exports the test models (smoke, the
 #     esp_nn:: mul, an ESP-NN delegate program and the Cadence-quantized wake
-#     word), generates and packs the component into <dir>/dist, and makes each
-#     model's kernel bundle, one for smoke and mul, and one for all four, all
-#     under <dir>/models. all_ops is the smoke model with a bundle that
+#     word), generates the component with build_idf_component.sh's stamp
+#     options (default --allow-dev-stamp), packs it into <dir>/dist, and makes
+#     each model's kernel bundle, one for smoke and mul, and one for all four,
+#     all under <dir>/models. all_ops is the smoke model with a bundle that
 #     registers every op of every kernel library.
 #   test_espressif_idf_nightly.sh build <dir> <shard> <shards>
 #     Needs only <dir> from prepare: installs ESPRESSIF_IDF_VERSION's ESP-IDF
@@ -21,14 +22,25 @@
 #     idf-build-apps (sdkconfig.ci.* times every target, filtered by
 #     .build-test-rules.yml), builds shard <shard> of <shards> with ESP-IDF's
 #     own Python environment, checks that each configuration's settings took
-#     effect, and runs each build on its emulator unless the rules disable its
-#     test. Reports every failure, then fails.
+#     effect, runs each build on its emulator unless the rules disable its
+#     test, and checks that the mirror served the packed archive's files.
+#     Reports every failure, then fails.
+#   test_espressif_idf_nightly.sh fetch <dir> <URL> <sha256>
+#     Extracts into <dir> the tar of prepare's version, dist and models that
+#     the publish workflow keeps at <URL>, if its hash is <sha256>.
+#   test_espressif_idf_nightly.sh registry <dir> <registry URL>
+#     Needs only <dir> from prepare: installs ESP-IDF 6.1, waits up to 15
+#     minutes for the given component registry to list <dir>'s version of
+#     pytorch/executorch, builds the smoke consumer for ESP32-S3 with it, fails
+#     if the registry serves files other than <dir>'s archive, and runs the
+#     build on the emulator.
 
 set -euo pipefail
 
 et_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
-mode=${1:?usage: $0 prepare <dir> | build <dir> <shard> <shards>}
-dir=${2:?usage: $0 prepare <dir> | build <dir> <shard> <shards>}
+usage="usage: $0 prepare <dir> [stamp options] | build <dir> <shard> <shards> | fetch <dir> <URL> <sha256> | registry <dir> <URL>"
+mode=${1:?${usage}}
+dir=${2:?${usage}}
 mkdir -p "${dir}"
 dir=$(cd "${dir}" && pwd)
 cd "${et_root}"
@@ -46,6 +58,12 @@ logged() {
 }
 
 prepare() {
+    local stamp=("${@:3}")
+    if (( ${#stamp[@]} == 0 )); then
+        stamp=(--allow-dev-stamp)
+    fi
+    # Checks the stamp against git before the model exports.
+    bash examples/espressif/build_idf_component.sh --version "${stamp[@]}" > /dev/null
     export ESPRESSIF_TOOLS_DIR="${ESPRESSIF_TOOLS_DIR:-${RUNNER_TEMP:-/tmp}/executorch-espressif}"
     local logs="${dir}/logs" models="${dir}/models"
     mkdir -p "${logs}" "${models}"
@@ -76,11 +94,13 @@ print(entry["atol"], entry["rtol"])
     rm -r "${models}"/*.bpte "${dir}/espnn"
 
     local component_dir="${dir}/component/executorch"
-    logged "${logs}/generate.log" bash examples/espressif/build_idf_component.sh \
-        --allow-dev-stamp --out "${component_dir}"
+    logged "${logs}/generate.log" bash examples/espressif/build_idf_component.sh "${stamp[@]}" \
+        --out "${component_dir}"
     local version
     version=$(sed -n 's/^version: "\(.*\)"$/\1/p' "${component_dir}/idf_component.yml")
-    (cd "${component_dir}" && compote component pack --name executorch --dest-dir "${dir}/dist")
+    # The registry links each version to its commit and to the generator.
+    (cd "${component_dir}" && compote component pack --name executorch --dest-dir "${dir}/dist" \
+        --commit-sha "$(git -C "${et_root}" rev-parse HEAD)" --repository-path examples/espressif)
     local model
     for model in smoke mul delegate wakeword; do
         python "${component_dir}/tools/gen_kernels.py" --pte "${models}/${model}.pte" \
@@ -114,28 +134,9 @@ build() {
     mkdir -p "${work}"
     work=$(cd "${work}" && pwd)
 
-    # ESP-IDF's QEMU needs these libraries, and setup.sh fails its install check
-    # without them.
-    if ! python -c '
-import ctypes
-ctypes.CDLL("libSDL2-2.0.so.0")
-ctypes.CDLL("libslirp.so.0")' 2> /dev/null; then
-        sudo apt-get update
-        sudo apt-get install -y --no-install-recommends libsdl2-2.0-0 libslirp0
-    fi
-    logged "${work}/setup.log" bash examples/espressif/setup.sh --minimal \
-        --idf-version="${idf_version}" --targets="$(IFS=,; echo "${targets[*]}")"
-    # shellcheck source=/dev/null
-    source "${ESPRESSIF_TOOLS_DIR}/setup_path.sh"
-    # A branch such as master moves, so record the commit a failure is against.
-    echo "ESP-IDF ${idf_version} at $(git -C "${IDF_PATH}" log -1 --format='%H (%cs)')" \
-        | tee "${work}/idf_commit.log"
-
-    # A fresh ESP-IDF Python environment, which cannot import executorch, and
-    # idf-build-apps in a venv of its own, so it changes nothing ESP-IDF runs.
-    export IDF_PYTHON_ENV_PATH="${work}/idf_python_env"
-    mkdir -p "${IDF_PYTHON_ENV_PATH}"
-    logged "${work}/idf_python_env.log" python "${IDF_PATH}/tools/idf_tools.py" install-python-env
+    install_idf "${idf_version}" "$(IFS=,; echo "${targets[*]}")" "${work}"
+    # Runs idf-build-apps from a venv of its own, so it changes nothing ESP-IDF
+    # runs.
     "${IDF_PYTHON_ENV_PATH}/bin/python" -m venv "${work}/idf_build_apps"
     "${work}/idf_build_apps/bin/python" -m pip install --quiet "idf-build-apps>=3.1,<4"
     local version
@@ -145,13 +146,8 @@ ctypes.CDLL("libslirp.so.0")' 2> /dev/null; then
         "${dir}/dist/executorch_${version}.tgz" "${work}/mirror" --namespace pytorch --name executorch
     export IDF_COMPONENT_STORAGE_URL="file://${work}/mirror;default"
 
-    # Built from a copy, so the component manager's state and the consumer's
-    # generated manifest stay out of the source tree.
     local sources="${work}/sources"
-    rm -rf "${sources}"
-    mkdir -p "${sources}/examples/espressif/idf_component_smoke"
-    cp -r examples/espressif/idf_component_consumer "${sources}/examples/espressif/"
-    cp -r examples/espressif/idf_component_smoke/main "${sources}/examples/espressif/idf_component_smoke/"
+    copy_consumer "${sources}"
     local app=examples/espressif/idf_component_consumer
     (cd "${sources}" && "${work}/idf_build_apps/bin/idf-build-apps" find -p "${app}" --target all \
         --default-build-targets "${targets[@]}" --config-rules 'sdkconfig.ci.*=' \
@@ -218,6 +214,11 @@ PY
             failures+=("${name} (run)")
         fi
     done 3< "${work}/shard.txt"
+    # Exercises the check that registry mode runs after an upload.
+    if ! served_matches "${dir}/dist/executorch_${version}.tgz" \
+        "${project}/managed_components/pytorch__executorch" "the mirror"; then
+        failures+=("served files")
+    fi
 
     if (( ${#failures[@]} )); then
         printf 'FAIL: %s\n' "${failures[@]}" >&2
@@ -226,6 +227,161 @@ PY
     echo "PASS: shard ${shard}/${shards} of the ESP-IDF ${idf_version} matrix built and ran."
 }
 
+# Installs ESP-IDF for the targets and makes a fresh ESP-IDF Python
+# environment, which cannot import executorch: ESP-IDF version, comma-separated
+# targets, work dir.
+install_idf() {
+    local idf_version=$1 targets_csv=$2 work=$3
+    # ESP-IDF's QEMU needs these libraries, and setup.sh fails its install check
+    # without them.
+    if [[ ",${targets_csv}," == *,esp32,* ]] && ! python -c '
+import ctypes
+ctypes.CDLL("libSDL2-2.0.so.0")
+ctypes.CDLL("libslirp.so.0")' 2> /dev/null; then
+        sudo apt-get update
+        sudo apt-get install -y --no-install-recommends libsdl2-2.0-0 libslirp0
+    fi
+    logged "${work}/setup.log" bash examples/espressif/setup.sh --minimal \
+        --idf-version="${idf_version}" --targets="${targets_csv}"
+    # shellcheck source=/dev/null
+    source "${ESPRESSIF_TOOLS_DIR}/setup_path.sh"
+    # A branch such as master moves, so record the commit a failure is against.
+    echo "ESP-IDF ${idf_version} at $(git -C "${IDF_PATH}" log -1 --format='%H (%cs)')" \
+        | tee "${work}/idf_commit.log"
+    export IDF_PYTHON_ENV_PATH="${work}/idf_python_env"
+    mkdir -p "${IDF_PYTHON_ENV_PATH}"
+    logged "${work}/idf_python_env.log" python "${IDF_PATH}/tools/idf_tools.py" install-python-env
+}
+
+# Copies the consumer project to a directory, so the component manager's state
+# and the consumer's generated manifest stay out of the source tree.
+copy_consumer() {
+    local sources=$1
+    rm -rf "${sources}"
+    mkdir -p "${sources}/examples/espressif/idf_component_smoke"
+    cp -r examples/espressif/idf_component_consumer "${sources}/examples/espressif/"
+    cp -r examples/espressif/idf_component_smoke/main "${sources}/examples/espressif/idf_component_smoke/"
+}
+
+fetch() {
+    local url=${3:?${usage}} sha256=${4:?${usage}} tar="${dir}/release.tar"
+    curl -sSf --retry 3 -o "${tar}" "${url}"
+    if ! echo "${sha256}  ${tar}" | sha256sum --check --quiet; then
+        echo "FAIL: ${url} is not the tested archive, whose sha256 is ${sha256}" >&2
+        exit 1
+    fi
+    tar -xf "${tar}" -C "${dir}"
+    rm "${tar}"
+}
+
+registry() {
+    local registry_url=${3:?${usage}}
+    local work="${ESPRESSIF_BUILD_DIR:-${et_root}/cmake-out-esp-idf-registry}"
+    export ESPRESSIF_TOOLS_DIR="${ESPRESSIF_TOOLS_DIR:-${RUNNER_TEMP:-/tmp}/executorch-espressif}"
+    mkdir -p "${work}"
+    work=$(cd "${work}" && pwd)
+    install_idf v6.1 esp32s3 "${work}"
+    local version
+    version=$(cat "${dir}/version")
+    local storage
+    if ! storage=$(curl -sSf --retry 3 "${registry_url%/}/api/" | python -c '
+import json, sys
+print(json.load(sys.stdin)["components_base_url"].rstrip("/"))'); then
+        echo "FAIL: ${registry_url}/api/ did not name the registry's storage" >&2
+        exit 1
+    fi
+    # The registry's storage is behind a CDN that caches for five minutes.
+    local tries=0
+    until curl -sSf "${storage}/components/pytorch/executorch.json" 2> /dev/null | python -c '
+import json, sys
+try:
+    versions = [v["version"] for v in json.load(sys.stdin)["versions"]]
+except (ValueError, KeyError):
+    versions = []
+sys.exit(sys.argv[1] not in versions)' "${version}"; do
+        if (( ++tries == 30 )); then
+            echo "FAIL: ${registry_url} did not list pytorch/executorch ${version}" >&2
+            exit 1
+        fi
+        echo "Waiting for ${registry_url} to list pytorch/executorch ${version}..."
+        sleep 30
+    done
+    copy_consumer "${work}/sources"
+    local project="${work}/sources/examples/espressif/idf_component_consumer"
+    local build="${work}/builds/esp32s3-default" log="${work}/emulator.log"
+    local idf_args=(-C "${project}" -B "${build}" -DSDKCONFIG="${build}/sdkconfig"
+        "-DSDKCONFIG_DEFAULTS=sdkconfig.defaults;sdkconfig.ci.default"
+        -DET_COMPONENT_VERSION="${version}" -DET_CONSUMER_MODELS_DIR="${dir}/models"
+        -DET_COMPONENT_REGISTRY_URL="${registry_url}")
+    IDF_TARGET=esp32s3 logged "${work}/build.log" "${IDF_PYTHON_ENV_PATH}/bin/python" \
+        "${IDF_PATH}/tools/idf.py" "${idf_args[@]}" build
+    # Checks that the registry serves the tested files: an --allow-existing
+    # upload leaves in place a version the registry already had.
+    served_matches "${dir}/dist/executorch_${version}.tgz" \
+        "${project}/managed_components/pytorch__executorch" "${registry_url}"
+    local model
+    model=$(sed -n 's/^CONFIG_ET_CONSUMER_MODEL="\(.*\)"$/\1/p' "${build}/sdkconfig")
+    run_firmware "${build}" esp32s3 "${log}" "${idf_args[@]}"
+    if ! grep -qE "${model}: PASS, [1-9][0-9]*/" "${log}" \
+        || ! grep -qF "ET_COMPONENT_SMOKE: PASS" "${log}"; then
+        tail -n 30 "${log}" >&2
+        echo "FAIL: the consumer from ${registry_url} did not pass on the emulator" >&2
+        exit 1
+    fi
+    echo "PASS: pytorch/executorch ${version} from ${registry_url} built and ran."
+}
+
+# Checks that a project got the files of the tested archive from a registry or
+# mirror. The component manager adds .component_hash and CHECKSUMS.json to what
+# it downloads, and the registry rewrites every idf_component.yml, the
+# component's and each example's, with sorted keys and no comments, and drops
+# the examples' override_path, so manifests are compared as YAML without
+# override_path.
+# Arguments: the archive, the project's managed_components/pytorch__executorch,
+# then the registry or mirror's name.
+served_matches() {
+    python - "$@" <<'PY'
+import hashlib
+import sys
+import tarfile
+from pathlib import Path
+
+import yaml
+
+
+def fingerprint(path, data):
+    if path == "idf_component.yml" or path.endswith("/idf_component.yml"):
+        manifest = yaml.safe_load(data)
+        for dependency in (manifest.get("dependencies") or {}).values():
+            if isinstance(dependency, dict):
+                dependency.pop("override_path", None)
+        return manifest
+    return hashlib.sha256(data).hexdigest()
+
+
+tested = {}
+with tarfile.open(sys.argv[1]) as archive:
+    for member in archive:
+        if member.isfile():
+            path = Path(member.name).as_posix()
+            tested[path] = fingerprint(path, archive.extractfile(member).read())
+served_dir = Path(sys.argv[2])
+served = {}
+for file in served_dir.rglob("*"):
+    if file.is_file():
+        path = file.relative_to(served_dir).as_posix()
+        served[path] = fingerprint(path, file.read_bytes())
+differ = sorted(
+    p
+    for p in tested.keys() | served.keys()
+    if p not in (".component_hash", "CHECKSUMS.json") and tested.get(p) != served.get(p)
+)
+if differ:
+    sys.exit(f"FAIL: {sys.argv[3]} served files unlike the tested archive's: {differ[:10]}")
+print(f"OK: {sys.argv[3]} served the tested archive's files")
+PY
+}
+
 # Checks that every setting of a sdkconfig.ci.* file reached the build's
 # sdkconfig, since ESP-IDF only warns about a symbol it does not know.
 # Arguments: the sdkconfig.ci.* file, then the build's sdkconfig.
@@ -274,7 +430,9 @@ run_firmware() {
 }
 
 case "${mode}" in
-    prepare) prepare ;;
+    prepare) prepare "$@" ;;
     build) build "$@" ;;
-    *) echo "unknown mode ${mode}; use prepare or build" >&2; exit 1 ;;
+    fetch) fetch "$@" ;;
+    registry) registry "$@" ;;
+    *) echo "unknown mode ${mode}; ${usage}" >&2; exit 1 ;;
 esac

.ci/scripts/test_espressif_idf_nightly.sh

@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Uploads the component archive that test_espressif_idf_nightly.sh prepare made
+# to an ESP Component Registry as pytorch/executorch, and succeeds if the
+# registry already has that version.
+#
+#   upload_espressif_idf_component.sh <dir> <registry URL> [--dry-run]
+#
+# Without --dry-run the job needs `id-token: write`: like
+# espressif/upload-components-ci-action, this authenticates with a GitHub OIDC
+# token for the registry, which accepts it when the component's trusted
+# uploader names this repository and the job's environment. --dry-run
+# validates the archive and needs no token.
+
+set -euo pipefail
+
+usage="usage: $0 <dir> <registry URL> [--dry-run]"
+if (( $# < 2 || $# > 3 )) || [[ $# == 3 && "$3" != --dry-run ]]; then
+    echo "${usage}" >&2
+    exit 1
+fi
+dir=$1 registry=$2 dry_run=("${@:3}")
+version=$(cat "${dir}/version")
+
+# Wheels only: pip would install an sdist's build dependencies unhashed, in a
+# job that can mint a registry token.
+python -m pip install --progress-bar off --require-hashes --only-binary :all: \
+    -r "$(dirname "${BASH_SOURCE[0]}")/espressif_idf_upload_requirements.txt"
+export IDF_COMPONENT_REGISTRY_URL="${registry}"
+if (( ${#dry_run[@]} == 0 )); then
+    token=$(curl -sSfG --data-urlencode "audience=${registry}" \
+        -H "Authorization: Bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN:?needs id-token: write}" \
+        "${ACTIONS_ID_TOKEN_REQUEST_URL}" | jq -er .value)
+    echo "::add-mask::${token}"
+    export IDF_COMPONENT_API_TOKEN="${token}"
+fi
+# --allow-existing skips a version the registry already has without a word,
+# so a re-run after an upload, or a dry run of a published version, validates
+# nothing; say so.
+if listing=$(curl -sSf "${registry%/}/api/components/pytorch/executorch" 2> /dev/null) \
+    && jq -e --arg version "${version}" 'any(.versions[]; .version == $version)' \
+        <<< "${listing}" > /dev/null; then
+    echo "::notice::pytorch/executorch ${version} is already on ${registry}, so compote skips it."
+fi
+compote component upload --namespace pytorch --name executorch \
+    --archive "${dir}/dist/executorch_${version}.tgz" --allow-existing "${dry_run[@]}"