Publish the ESP-IDF component to the registry

publish-espressif-idf.yml publishes pytorch/executorch to the ESP
Component Registry. An RC tag vX.Y.Z-rcN publishes X.Y.Z-rc.N to the
staging registry, a release tag vX.Y.Z publishes X.Y.Z to staging and
then production, and a respin tag vX.Y.Z-esp-idf.N, which the generator
requires for X.Y.Z~N since git refs cannot hold ~, publishes that respin
the same way. Other v* tags publish nothing.

The workflow packs the component once, under the generator's
release-stamp checks and with the compiler cache off, so the flatc that
generates the archive's headers and the executorch that exports its
example's model are built fresh, and asserts the version matches the
tag. Before anything is uploaded, it builds and runs that archive in
every consumer configuration on ESP-IDF 6.0.1 and 6.1 and validates it
with both registries. Each upload waits for a reviewer of the
esp-component-registry environment. After each upload, the new registry
mode of test_espressif_idf_nightly.sh polls until the registry lists the
version, builds the consumer with pytorch/executorch resolved from that
registry, checks that the registry served the tested archive's files,
comparing every idf_component.yml as YAML, since the registry rewrites
the component's and each example's with sorted keys and no comments, and
without the examples' override_path, which it drops, and runs it on the
emulator. Production waits for that check on staging. The nightly's
build mode runs the same file check against the mirror, so it is
exercised before the first upload. The consumer takes the registry for
pytorch/executorch as -DET_COMPONENT_REGISTRY_URL.

The jobs after an approval cannot read the tested archive from the run's
GitHub artifact, which lasts 2 days, less than the approvals can take.
The package job puts the tested files in one tar, which every later job
reads, a job records the tar's sha256 as an output before any approval,
and another copies the tar to the run's S3 prefix in gha-artifacts,
where it lasts 14 days, under a name with that hash. The upload and
registry-check jobs fetch it over HTTPS and refuse it unless the hash
matches, since other jobs can write to that bucket. The build and
registry-check jobs keep their logs there too, named per job and
attempt, because an upload to a name already taken fails.

Uploads go through .ci/scripts/upload_espressif_idf_component.sh, which
the nightly's prepare job now also runs for its dry run, after running
fetch on a local copy of the tar, so a change to either is tested every
day. It installs the component manager from a hash-locked requirements
file, wheels only, and the two upload jobs pin their actions by commit,
since anything they run could mint a registry token. It authenticates
with the job's GitHub OIDC token for the registry, the way
espressif/upload-components-ci-action does, and uploads the archive that
was tested, with --allow-existing so a failed run can be rerun. A
version already on the registry is kept, so a rerun that packed
different bytes fails the registry check instead of shipping them. Since
compote then skips the upload and its validation silently, the script
says when the registry already has the version. prepare now records the
commit and generator in that archive at pack time, because compote
applies them only when it packs. Manual runs, which must be on a
release, RC or respin tag since the generator stamps those only at their
tag, take their version from that tag, so the approved ref always names
what is uploaded, and validate without uploading unless dry_run is
cleared, and every run that uploads nothing runs the staging upload job
with --dry-run outside the environment, so its runner, actions and
archive fetch work before a release needs them. Pull requests that touch
the workflow run it that way, stamping a release version at a throwaway
local tag, because workflow_dispatch cannot run before the workflow is
on main, and the nightly runs the same release-stamp checks every day,
so a change that would break them shows up before a real tag. Runs share
a concurrency group per tag or branch, so a run waiting for an approval
never cancels another tag's.

The manifest pins esp-nn to the production registry, because the staging
registry records a dependency without one as its own, and staging has no
esp-nn; the component manager treats that as the same source as an
unnamed dependency on the default registry, and the component's README
says where esp-nn comes from.

The README's runbook covers which rerun works when, release branches cut
before the workflow landed, the concurrency group, and the one-time
setup, in order: rulesets that leave v* tags, which the environment
trusts, and release/* branches, which respin tags must be on, to release
managers, then the protected environment, which deploys only from v*
tags, since GitHub creates an unprotected one as soon as a workflow
names it, then the staging namespace, which that registry grants
separately, then each registry's trusted uploader, which names
pytorch/executorch as the repository, publish-espressif-idf.yml as the
workflow and esp-component-registry as the environment, with no branch,
because tag pushes carry none. Reviewers check that a run's ref and
commit are the release's, or for a respin that its tag's commit is on
release/X.Y with reviewed changes; they gate the staging upload on the
nightly's macOS and Windows configure jobs, with ESP-IDF master
advisory, and approve the production upload only once executorch X.Y.Z
is on PyPI, which projects need to make kernel bundles for it. The first
release follows an RC that this workflow published, so the upload and
registry checks have run before a release depends on them.

Authored with Claude Code.

Author: rascani

.ci/scripts/esp_idf_mirror.py

@@ -68,7 +68,7 @@ def main() -> None:
             {
                 "spec": requirement.version_spec,
                 "source": requirement.source.type,
-                "registry_url": None,
+                "registry_url": requirement.registry_url,
                 "is_public": requirement.is_public,
                 "require": requirement.is_required,
                 "name": None if is_idf else name,

.ci/scripts/espressif_idf_upload_requirements.txt

@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Uploads the component archive that test_espressif_idf_nightly.sh prepare made
+# to an ESP Component Registry as pytorch/executorch, and succeeds if the
+# registry already has that version.
+#
+#   upload_espressif_idf_component.sh <dir> <registry URL> [--dry-run]
+#
+# Without --dry-run the job needs `id-token: write`: like
+# espressif/upload-components-ci-action, this authenticates with a GitHub OIDC
+# token for the registry, which accepts it when the component's trusted
+# uploader names this repository and the job's environment. --dry-run
+# validates the archive and needs no token.
+
+set -euo pipefail
+
+usage="usage: $0 <dir> <registry URL> [--dry-run]"
+if (( $# < 2 || $# > 3 )) || [[ $# == 3 && "$3" != --dry-run ]]; then
+    echo "${usage}" >&2
+    exit 1
+fi
+dir=$1 registry=$2 dry_run=("${@:3}")
+version=$(cat "${dir}/version")
+
+# Wheels only: pip would install an sdist's build dependencies unhashed, in a
+# job that can mint a registry token.
+python -m pip install --progress-bar off --require-hashes --only-binary :all: \
+    -r "$(dirname "${BASH_SOURCE[0]}")/espressif_idf_upload_requirements.txt"
+export IDF_COMPONENT_REGISTRY_URL="${registry}"
+if (( ${#dry_run[@]} == 0 )); then
+    token=$(curl -sSfG --data-urlencode "audience=${registry}" \
+        -H "Authorization: Bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN:?needs id-token: write}" \
+        "${ACTIONS_ID_TOKEN_REQUEST_URL:?needs id-token: write}" | jq -er .value)
+    echo "::add-mask::${token}"
+    export IDF_COMPONENT_API_TOKEN="${token}"
+fi
+# --allow-existing silently skips a version the registry already has, so a
+# re-run after an upload, or a dry run of a published version, validates
+# nothing; report it.
+if listing=$(curl -sSf "${registry%/}/api/components/pytorch/executorch" 2> /dev/null) \
+    && jq -e --arg version "${version}" 'any(.versions[]; .version == $version)' \
+        <<< "${listing}" > /dev/null; then
+    echo "::notice::pytorch/executorch ${version} is already on ${registry}, so compote skips it."
+fi
+compote component upload --namespace pytorch --name executorch \
+    --archive "${dir}/dist/executorch_${version}.tgz" --allow-existing "${dry_run[@]}"

.ci/scripts/test_espressif_idf_nightly.sh

@@ -50,12 +50,20 @@ jobs:
         source .ci/scripts/utils.sh
         install_executorch --use-pt-pinned-commit --minimal
         bash .ci/scripts/test_espressif_idf_nightly.sh prepare "${RUNNER_ARTIFACT_DIR}"
+        # Runs the release stamp's checks at a throwaway local tag, so they do
+        # not first run at a real one. The check of the executorch package,
+        # which --version stops before, runs only when publishing.
+        git tag -f "v$(cat version.txt)"
+        bash examples/espressif/build_idf_component.sh --version --version-suffix ""
+        # Exercises fetch, which otherwise runs only when publishing, on a
+        # local copy of the tar that publishing keeps.
+        tar -cf "${RUNNER_TEMP}/release.tar" -C "${RUNNER_ARTIFACT_DIR}" version dist models
+        bash .ci/scripts/test_espressif_idf_nightly.sh fetch "${RUNNER_TEMP}/fetched" \
+          "file://${RUNNER_TEMP}/release.tar" "$(sha256sum "${RUNNER_TEMP}/release.tar" | cut -d' ' -f1)"
         # The registry's validation of the packed archive, last, since it
         # installs the component manager into this environment.
-        pip install --progress-bar off "idf-component-manager~=3.1"
-        version=$(cat "${RUNNER_ARTIFACT_DIR}/version")
-        compote component upload --namespace pytorch --name executorch \
-          --archive "${RUNNER_ARTIFACT_DIR}/dist/executorch_${version}.tgz" --dry-run --allow-existing
+        bash .ci/scripts/upload_espressif_idf_component.sh "${RUNNER_ARTIFACT_DIR}" \
+          https://components.espressif.com --dry-run
 
   build:
     name: ESP-IDF ${{ matrix.idf-version }} (${{ matrix.shard }}/4)