Skip to content

Commit ce19cfc

Browse files
rascanirascani
committed
Publish the ESP-IDF component to the registry
publish-esp-idf.yml publishes pytorch/executorch to the ESP Component Registry. An RC tag vX.Y.Z-rcN publishes X.Y.Z-rc.N to the staging registry, a release tag vX.Y.Z publishes X.Y.Z to staging and then production, and a manual run on release/X.Y publishes a respin, X.Y.Z~N. Other v* tags publish nothing. The workflow packs the component once, under the generator's release-stamp checks, and asserts the version matches the tag. Before anything is uploaded, it builds and runs that archive in every consumer configuration on ESP-IDF 6.0.1 and 6.1 and validates it with both registries. Each upload waits for a reviewer of the esp-component-registry environment. After each upload, the new registry mode of test_espressif_idf_nightly.sh polls until the registry lists the version, builds the consumer with pytorch/executorch resolved from that registry, checks that the registry served the tested archive's files (apart from the examples' manifests, which the registry rewrites to drop override_path), and runs it on the emulator. Production waits for that check on staging. The nightly's build mode runs the same file check against the mirror, so it is exercised before the first upload. The consumer takes the registry as -DET_COMPONENT_REGISTRY_URL. Uploads go through .ci/scripts/upload_espressif_idf_component.sh, which the nightly's dry-run job now shares. It installs the component manager from a hash-locked requirements file, and the two upload jobs pin their actions by commit, since anything they run could mint a registry token. It authenticates with the job's GitHub OIDC token for the registry, the way espressif/upload-components-ci-action does, and uploads the archive that was tested, with --allow-existing so a failed run can be rerun. A version already on the registry is kept, so a rerun that packed different bytes fails the registry check instead of shipping them. prepare now records the commit and generator path in that archive at pack time, because compote applies them only when it packs. Manual runs validate without uploading unless dry_run is cleared. Pull requests that touch the workflow run it as a dry run that stamps a release version at a throwaway local tag, because workflow_dispatch cannot run before the workflow is on main, and the nightly runs the same release-stamp checks every day, so a change that would break them shows up before a real tag. Runs share a concurrency group per tag or branch, so a run waiting for an approval never cancels another tag's. The manifest names production as esp-nn's registry, because the staging registry records a dependency without one as its own, and staging has no esp-nn; the component manager treats that as the same source as an unnamed dependency on the default registry. The README's runbook covers the one-time setup, in order: the protected environment first, since GitHub creates an unprotected one as soon as a workflow names it, then the staging namespace, then each registry's trusted uploader, which names pytorch/executorch as the repository and esp-component-registry as the environment, with no branch, because the registry cannot pin a workflow and tag pushes carry no branch. Authored with Claude Code.
1 parent ed7bb36 commit ce19cfc

9 files changed

Lines changed: 1028 additions & 45 deletions

File tree

‎.ci/scripts/esp_idf_mirror.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ def main() -> None:
6868
{
6969
"spec": requirement.version_spec,
7070
"source": requirement.source.type,
71-
"registry_url": None,
71+
"registry_url": requirement.registry_url,
7272
"is_public": requirement.is_public,
7373
"require": requirement.is_required,
7474
"name": None if is_idf else name,

‎.ci/scripts/espressif_idf_upload_requirements.txt‎

Lines changed: 460 additions & 0 deletions
Large diffs are not rendered by default.

‎.ci/scripts/test_espressif_idf_nightly.sh‎

Lines changed: 164 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,15 @@
55
# This source code is licensed under the BSD-style license found in the
66
# LICENSE file in the root directory of this source tree.
77

8-
# The ExecuTorch ESP-IDF component's nightly matrix.
8+
# The ExecuTorch ESP-IDF component's nightly matrix and release checks.
99
#
10-
# test_espressif_idf_nightly.sh prepare <dir>
10+
# test_espressif_idf_nightly.sh prepare <dir> [stamp options]
1111
# In the ExecuTorch environment: exports the test models (smoke, the
1212
# esp_nn:: mul, an ESP-NN delegate program and the Cadence-quantized wake
13-
# word), generates and packs the component into a file:// component mirror,
14-
# and makes each model's kernel bundle, all under <dir>. all_ops is the
13+
# word), generates the component with build_idf_component.sh's stamp
14+
# options (default --allow-dev-stamp), packs it into <dir>/dist and a
15+
# file:// component mirror, and makes each model's kernel bundle, all under
16+
# <dir>. all_ops is the
1517
# smoke model with a bundle that registers every op of every kernel
1618
# library.
1719
# test_espressif_idf_nightly.sh build <dir> <shard> <shards>
@@ -22,12 +24,19 @@
2224
# own Python environment, checks that each configuration's settings took
2325
# effect, and runs each build on its emulator unless the rules disable its
2426
# test. Reports every failure, then fails.
27+
# test_espressif_idf_nightly.sh registry <dir> <registry URL>
28+
# Needs only <dir> from prepare: installs ESP-IDF 6.1, waits up to 15
29+
# minutes for the given component registry to list <dir>'s version of
30+
# pytorch/executorch, builds the smoke consumer for ESP32-S3 with it, fails
31+
# if the registry serves files other than <dir>'s archive, and runs the
32+
# build on the emulator.
2533

2634
set -euo pipefail
2735

2836
et_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
29-
mode=${1:?usage: $0 prepare <dir> | build <dir> <shard> <shards>}
30-
dir=${2:?usage: $0 prepare <dir> | build <dir> <shard> <shards>}
37+
usage="usage: $0 prepare <dir> [stamp options] | build <dir> <shard> <shards> | registry <dir> <URL>"
38+
mode=${1:?${usage}}
39+
dir=${2:?${usage}}
3140
mkdir -p "${dir}"
3241
dir=$(cd "${dir}" && pwd)
3342
cd "${et_root}"
@@ -45,6 +54,12 @@ logged() {
4554
}
4655

4756
prepare() {
57+
local stamp=("${@:3}")
58+
if (( ${#stamp[@]} == 0 )); then
59+
stamp=(--allow-dev-stamp)
60+
fi
61+
# Checks the stamp against git before the model exports.
62+
bash examples/espressif/build_idf_component.sh --version "${stamp[@]}" > /dev/null
4863
export ESPRESSIF_TOOLS_DIR="${ESPRESSIF_TOOLS_DIR:-${RUNNER_TEMP:-/tmp}/executorch-espressif}"
4964
local logs="${dir}/logs" models="${dir}/models"
5065
mkdir -p "${logs}" "${models}"
@@ -74,11 +89,13 @@ print(entry["atol"], entry["rtol"])
7489
rm -r "${models}"/*.bpte "${dir}/espnn"
7590

7691
local component_dir="${dir}/component/executorch"
77-
logged "${logs}/generate.log" bash examples/espressif/build_idf_component.sh \
78-
--allow-dev-stamp --out "${component_dir}"
92+
logged "${logs}/generate.log" bash examples/espressif/build_idf_component.sh "${stamp[@]}" \
93+
--out "${component_dir}"
7994
local version
8095
version=$(sed -n 's/^version: "\(.*\)"$/\1/p' "${component_dir}/idf_component.yml")
81-
(cd "${component_dir}" && compote component pack --name executorch --dest-dir "${dir}/dist")
96+
# The registry links each version to its commit and to the generator.
97+
(cd "${component_dir}" && compote component pack --name executorch --dest-dir "${dir}/dist" \
98+
--commit-sha "$(git -C "${et_root}" rev-parse HEAD)" --repository-path examples/espressif)
8299
python .ci/scripts/esp_idf_mirror.py "${dir}/dist/executorch_${version}.tgz" "${dir}/mirror" \
83100
--namespace pytorch --name executorch
84101
local model
@@ -111,38 +128,16 @@ build() {
111128
mkdir -p "${work}"
112129
work=$(cd "${work}" && pwd)
113130

114-
# ESP-IDF's QEMU needs these libraries, and setup.sh fails its install check
115-
# without them.
116-
if ! python -c '
117-
import ctypes
118-
ctypes.CDLL("libSDL2-2.0.so.0")
119-
ctypes.CDLL("libslirp.so.0")' 2> /dev/null; then
120-
sudo apt-get update
121-
sudo apt-get install -y --no-install-recommends libsdl2-2.0-0 libslirp0
122-
fi
123-
logged "${work}/setup.log" bash examples/espressif/setup.sh --minimal \
124-
--idf-version="${idf_version}" --targets="$(IFS=,; echo "${targets[*]}")"
125-
# shellcheck source=/dev/null
126-
source "${ESPRESSIF_TOOLS_DIR}/setup_path.sh"
127-
128-
# A fresh ESP-IDF Python environment, which cannot import executorch, and
131+
install_idf "${idf_version}" "$(IFS=,; echo "${targets[*]}")" "${work}"
129132
# idf-build-apps in a venv of its own, so it changes nothing ESP-IDF runs.
130-
export IDF_PYTHON_ENV_PATH="${work}/idf_python_env"
131-
mkdir -p "${IDF_PYTHON_ENV_PATH}"
132-
logged "${work}/idf_python_env.log" python "${IDF_PATH}/tools/idf_tools.py" install-python-env
133133
"${IDF_PYTHON_ENV_PATH}/bin/python" -m venv "${work}/idf_build_apps"
134134
"${work}/idf_build_apps/bin/python" -m pip install --quiet "idf-build-apps>=3.1,<4"
135135
export IDF_COMPONENT_STORAGE_URL="file://${dir}/mirror;default"
136136
local version
137137
version=$(cat "${dir}/version")
138138

139-
# Built from a copy, so the component manager's state and the consumer's
140-
# generated manifest stay out of the source tree.
141139
local sources="${work}/sources"
142-
rm -rf "${sources}"
143-
mkdir -p "${sources}/examples/espressif/idf_component_smoke"
144-
cp -r examples/espressif/idf_component_consumer "${sources}/examples/espressif/"
145-
cp -r examples/espressif/idf_component_smoke/main "${sources}/examples/espressif/idf_component_smoke/"
140+
copy_consumer "${sources}"
146141
local app=examples/espressif/idf_component_consumer
147142
(cd "${sources}" && "${work}/idf_build_apps/bin/idf-build-apps" find -p "${app}" --target all \
148143
--default-build-targets "${targets[@]}" --config-rules 'sdkconfig.ci.*=' \
@@ -199,6 +194,12 @@ PY
199194
failures+=("${name} (run)")
200195
fi
201196
done 3< "${work}/shard.txt"
197+
# Exercises the check that registry mode runs after an upload.
198+
if [[ -d "${project}/managed_components/pytorch__executorch" ]] \
199+
&& ! served_matches "${dir}/dist/executorch_${version}.tgz" \
200+
"${project}/managed_components/pytorch__executorch" "the mirror"; then
201+
failures+=("served files")
202+
fi
202203

203204
if (( ${#failures[@]} )); then
204205
printf 'FAIL: %s\n' "${failures[@]}" >&2
@@ -207,6 +208,133 @@ PY
207208
echo "PASS: shard ${shard}/${shards} of the ESP-IDF ${idf_version} matrix built and ran."
208209
}
209210

211+
# Installs ESP-IDF for the targets and makes a fresh ESP-IDF Python
212+
# environment, which cannot import executorch: ESP-IDF version, comma-separated
213+
# targets, work dir.
214+
install_idf() {
215+
local idf_version=$1 targets_csv=$2 work=$3
216+
# ESP-IDF's QEMU needs these libraries, and setup.sh fails its install check
217+
# without them.
218+
if [[ ",${targets_csv}," == *,esp32,* ]] && ! python -c '
219+
import ctypes
220+
ctypes.CDLL("libSDL2-2.0.so.0")
221+
ctypes.CDLL("libslirp.so.0")' 2> /dev/null; then
222+
sudo apt-get update
223+
sudo apt-get install -y --no-install-recommends libsdl2-2.0-0 libslirp0
224+
fi
225+
logged "${work}/setup.log" bash examples/espressif/setup.sh --minimal \
226+
--idf-version="${idf_version}" --targets="${targets_csv}"
227+
# shellcheck source=/dev/null
228+
source "${ESPRESSIF_TOOLS_DIR}/setup_path.sh"
229+
export IDF_PYTHON_ENV_PATH="${work}/idf_python_env"
230+
mkdir -p "${IDF_PYTHON_ENV_PATH}"
231+
logged "${work}/idf_python_env.log" python "${IDF_PATH}/tools/idf_tools.py" install-python-env
232+
}
233+
234+
# Copies the consumer project to a directory, so the component manager's state
235+
# and the consumer's generated manifest stay out of the source tree.
236+
copy_consumer() {
237+
local sources=$1
238+
rm -rf "${sources}"
239+
mkdir -p "${sources}/examples/espressif/idf_component_smoke"
240+
cp -r examples/espressif/idf_component_consumer "${sources}/examples/espressif/"
241+
cp -r examples/espressif/idf_component_smoke/main "${sources}/examples/espressif/idf_component_smoke/"
242+
}
243+
244+
registry() {
245+
local registry_url=${3:?${usage}}
246+
local work="${ESPRESSIF_BUILD_DIR:-${et_root}/cmake-out-esp-idf-registry}"
247+
export ESPRESSIF_TOOLS_DIR="${ESPRESSIF_TOOLS_DIR:-${RUNNER_TEMP:-/tmp}/executorch-espressif}"
248+
mkdir -p "${work}"
249+
work=$(cd "${work}" && pwd)
250+
install_idf v6.1 esp32s3 "${work}"
251+
local version
252+
version=$(cat "${dir}/version")
253+
local storage
254+
if ! storage=$(curl -sSf --retry 3 "${registry_url%/}/api/" | python -c '
255+
import json, sys
256+
print(json.load(sys.stdin)["components_base_url"].rstrip("/"))'); then
257+
echo "FAIL: ${registry_url}/api/ did not name the registry's storage" >&2
258+
exit 1
259+
fi
260+
# The registry's storage is behind a CDN that caches for five minutes.
261+
local tries=0
262+
until curl -sSf "${storage}/components/pytorch/executorch.json" 2> /dev/null | python -c '
263+
import json, sys
264+
try:
265+
versions = [v["version"] for v in json.load(sys.stdin)["versions"]]
266+
except ValueError:
267+
versions = []
268+
sys.exit(sys.argv[1] not in versions)' "${version}"; do
269+
if (( ++tries == 30 )); then
270+
echo "FAIL: ${registry_url} did not list pytorch/executorch ${version}" >&2
271+
exit 1
272+
fi
273+
echo "Waiting for ${registry_url} to list pytorch/executorch ${version}..."
274+
sleep 30
275+
done
276+
copy_consumer "${work}/sources"
277+
local project="${work}/sources/examples/espressif/idf_component_consumer"
278+
local build="${work}/builds/esp32s3-default" log="${work}/emulator.log"
279+
local idf_args=(-C "${project}" -B "${build}" -DSDKCONFIG="${build}/sdkconfig"
280+
"-DSDKCONFIG_DEFAULTS=sdkconfig.defaults;sdkconfig.ci.default"
281+
-DET_COMPONENT_VERSION="${version}" -DET_CONSUMER_MODELS_DIR="${dir}/models"
282+
-DET_COMPONENT_REGISTRY_URL="${registry_url}")
283+
IDF_TARGET=esp32s3 logged "${work}/build.log" "${IDF_PYTHON_ENV_PATH}/bin/python" \
284+
"${IDF_PATH}/tools/idf.py" "${idf_args[@]}" build
285+
# --allow-existing uploads would not replace an archive already there.
286+
served_matches "${dir}/dist/executorch_${version}.tgz" \
287+
"${project}/managed_components/pytorch__executorch" "${registry_url}"
288+
local model
289+
model=$(sed -n 's/^CONFIG_ET_CONSUMER_MODEL="\(.*\)"$/\1/p' "${build}/sdkconfig")
290+
run_firmware "${build}" esp32s3 "${log}" "${idf_args[@]}"
291+
if ! grep -qF "${model}: PASS" "${log}" || ! grep -qF "ET_COMPONENT_SMOKE: PASS" "${log}"; then
292+
tail -n 30 "${log}" >&2
293+
echo "FAIL: the consumer from ${registry_url} did not pass on the emulator" >&2
294+
exit 1
295+
fi
296+
echo "PASS: pytorch/executorch ${version} from ${registry_url} built and ran."
297+
}
298+
299+
# Checks that a project got the files of the tested archive from a registry or
300+
# mirror: archive, the project's managed_components/pytorch__executorch, source.
301+
# The registry rewrites the examples' manifests, dropping override_path, so
302+
# those are the only files that may differ.
303+
served_matches() {
304+
python - "$@" <<'PY'
305+
import hashlib
306+
import sys
307+
import tarfile
308+
from pathlib import Path
309+
310+
311+
def compared(path):
312+
return path not in (".component_hash", "CHECKSUMS.json") and not (
313+
path.startswith("examples/") and path.endswith("/idf_component.yml")
314+
)
315+
316+
317+
with tarfile.open(sys.argv[1]) as archive:
318+
tested = {
319+
Path(m.name).as_posix(): hashlib.sha256(archive.extractfile(m).read()).hexdigest()
320+
for m in archive
321+
if m.isfile()
322+
}
323+
served_dir = Path(sys.argv[2])
324+
served = {
325+
p.relative_to(served_dir).as_posix(): hashlib.sha256(p.read_bytes()).hexdigest()
326+
for p in served_dir.rglob("*")
327+
if p.is_file()
328+
}
329+
differ = sorted(
330+
p for p in tested.keys() | served.keys() if compared(p) and tested.get(p) != served.get(p)
331+
)
332+
if differ:
333+
sys.exit(f"FAIL: {sys.argv[3]} served files unlike the tested archive's: {differ[:10]}")
334+
print(f"OK: {sys.argv[3]} served the tested archive's files")
335+
PY
336+
}
337+
210338
# Checks that every setting of a sdkconfig.ci.* file reached the build's
211339
# sdkconfig, since ESP-IDF only warns about a symbol it does not know:
212340
# defaults file, sdkconfig.
@@ -255,7 +383,8 @@ run_firmware() {
255383
}
256384

257385
case "${mode}" in
258-
prepare) prepare ;;
386+
prepare) prepare "$@" ;;
259387
build) build "$@" ;;
260-
*) echo "unknown mode ${mode}; use prepare or build" >&2; exit 1 ;;
388+
registry) registry "$@" ;;
389+
*) echo "unknown mode ${mode}; ${usage}" >&2; exit 1 ;;
261390
esac

‎.ci/scripts/upload_espressif_idf_component.sh‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/usr/bin/env bash
2+
# Copyright (c) Meta Platforms, Inc. and affiliates.
3+
# All rights reserved.
4+
#
5+
# This source code is licensed under the BSD-style license found in the
6+
# LICENSE file in the root directory of this source tree.
7+
8+
# Uploads the component archive that test_espressif_idf_nightly.sh prepare made
9+
# to an ESP Component Registry as pytorch/executorch, and succeeds if the
10+
# registry already has that version.
11+
#
12+
# upload_espressif_idf_component.sh <dir> <registry URL> [--dry-run]
13+
#
14+
# Without --dry-run the job needs `id-token: write`: like
15+
# espressif/upload-components-ci-action, this authenticates with a GitHub OIDC
16+
# token for the registry, which accepts it when the component's trusted
17+
# uploader names this repository and the job's environment. --dry-run
18+
# validates the archive and needs no token.
19+
20+
set -euo pipefail
21+
22+
usage="usage: $0 <dir> <registry URL> [--dry-run]"
23+
if (( $# < 2 || $# > 3 )) || [[ $# == 3 && "$3" != --dry-run ]]; then
24+
echo "${usage}" >&2
25+
exit 1
26+
fi
27+
dir=$1 registry=$2 dry_run=("${@:3}")
28+
version=$(cat "${dir}/version")
29+
30+
python -m pip install --progress-bar off --require-hashes \
31+
-r "$(dirname "${BASH_SOURCE[0]}")/espressif_idf_upload_requirements.txt"
32+
export IDF_COMPONENT_REGISTRY_URL="${registry}"
33+
if (( ${#dry_run[@]} == 0 )); then
34+
token=$(curl -sSfG --data-urlencode "audience=${registry}" \
35+
-H "Authorization: Bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN:?needs id-token: write}" \
36+
"${ACTIONS_ID_TOKEN_REQUEST_URL}" | jq -er .value)
37+
echo "::add-mask::${token}"
38+
export IDF_COMPONENT_API_TOKEN="${token}"
39+
fi
40+
compote component upload --namespace pytorch --name executorch \
41+
--archive "${dir}/dist/executorch_${version}.tgz" --allow-existing "${dry_run[@]}"

‎.github/workflows/nightly-espressif-idf.yml‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@ on:
1313
- .ci/scripts/test_espressif_idf_nightly.sh
1414
- .ci/scripts/test_espressif_idf_configure.py
1515
- .ci/scripts/esp_idf_mirror.py
16+
- .ci/scripts/upload_espressif_idf_component.sh
17+
- .ci/scripts/espressif_idf_upload_requirements.txt
1618
- examples/espressif/build_idf_component.sh
1719
- examples/espressif/setup.sh
1820
- examples/espressif/export_smoke_model.py
@@ -55,6 +57,10 @@ jobs:
5557
source .ci/scripts/utils.sh
5658
install_executorch --use-pt-pinned-commit --minimal
5759
bash .ci/scripts/test_espressif_idf_nightly.sh prepare "${RUNNER_ARTIFACT_DIR}"
60+
# The release stamp's checks, which publishing otherwise first runs at a
61+
# real tag, at a throwaway local one.
62+
git tag -f "v$(cat version.txt)"
63+
bash examples/espressif/build_idf_component.sh --version --version-suffix ""
5864
5965
registry-dry-run:
6066
name: Registry validation (upload --dry-run)
@@ -73,10 +79,8 @@ jobs:
7379
set -euo pipefail
7480
CONDA_ENV=$(conda env list --json | jq -r '.envs | .[-1]')
7581
conda activate "${CONDA_ENV}"
76-
pip install --progress-bar off "idf-component-manager~=3.1"
77-
version=$(cat "${RUNNER_ARTIFACT_DIR}/version")
78-
compote component upload --namespace pytorch --name executorch \
79-
--archive "${RUNNER_ARTIFACT_DIR}/dist/executorch_${version}.tgz" --dry-run --allow-existing
82+
bash .ci/scripts/upload_espressif_idf_component.sh "${RUNNER_ARTIFACT_DIR}" \
83+
https://components.espressif.com --dry-run
8084
8185
build:
8286
name: ESP-IDF ${{ matrix.idf-version }} (${{ matrix.shard }}/4)

0 commit comments

Comments
 (0)