Skip to content

Commit f41530e

Browse files
rascanirascani
committed
Publish the ESP-IDF component to the registry
publish-esp-idf.yml publishes pytorch/executorch to the ESP Component Registry. An RC tag vX.Y.Z-rcN publishes X.Y.Z-rc.N to the staging registry, a release tag vX.Y.Z publishes X.Y.Z to staging and then production, and a manual run on release/X.Y publishes a packaging respin, X.Y.Z~N. Other v* tags publish nothing. The workflow packs the component once, under the generator's release-stamp checks, and asserts the version matches the tag. Before anything is uploaded, it builds and runs that archive in every consumer configuration on ESP-IDF 6.0.1 and 6.1 and validates it with the registry. Each upload waits for a reviewer of the esp-component-registry environment. After each upload, the new registry mode of test_espressif_idf_nightly.sh polls until the registry lists the version, builds the consumer with pytorch/executorch resolved from that registry, checks that the registry served the tested archive's files (apart from the examples' manifests, which the registry rewrites to drop override_path), and runs it on the emulator. Production waits for that check on staging. The consumer takes the registry from ET_COMPONENT_REGISTRY_URL. Uploads go through .ci/scripts/upload_espressif_idf_component.sh, which the nightly's dry-run job now shares. It authenticates with the job's GitHub OIDC token for the registry, the way espressif/upload-components-ci-action does, and uploads the archive that was tested, with --allow-existing so a failed run can be rerun. A version already on the registry is kept, so a rerun that packed different bytes fails the registry check instead of shipping them. prepare now records the commit and generator path in that archive at pack time, because compote applies them only when it packs. Manual runs validate without uploading unless dry_run is cleared. Pull requests that touch the workflow run it as a dry run that stamps a release version at a throwaway local tag, because workflow_dispatch cannot run before the workflow is on main and the release stamp's clean-tree check would otherwise first run on a real tag. Publishing runs share one concurrency group, so two runs never upload the same version at once. A respin may now also change the component's CI (its .ci/scripts and workflows) since the release tag, so a release whose publishing broke can be published as X.Y.Z~1 once the pipeline is fixed. The manifest names production as esp-nn's registry, because the staging registry records dependencies without one as its own, and staging has no esp-nn. The first upload needs these setup steps. Staging needs the pytorch namespace and pytorch/executorch, which production already has. On each registry, pytorch/executorch's trusted uploader names pytorch/executorch as the repository and esp-component-registry as the environment, with no branch, because the registry cannot pin a workflow and tag pushes carry no branch. A repo admin creates that environment with reviewers, self-review prevented, no admin bypass, and deployments limited to v* tags and release/* branches. Approve the staging upload once the nightly matrix is green on the release commit, and the production upload once the version on staging has been checked. Authored with Claude Code.
1 parent db7697c commit f41530e

9 files changed

Lines changed: 537 additions & 54 deletions

File tree

‎.ci/scripts/test_espressif_idf_nightly.sh‎

Lines changed: 149 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -5,26 +5,33 @@
55
# This source code is licensed under the BSD-style license found in the
66
# LICENSE file in the root directory of this source tree.
77

8-
# The ExecuTorch ESP-IDF component's nightly matrix.
8+
# The ExecuTorch ESP-IDF component's nightly matrix and release checks.
99
#
10-
# test_espressif_idf_nightly.sh prepare <dir>
10+
# test_espressif_idf_nightly.sh prepare <dir> [stamp options]
1111
# In the ExecuTorch environment: exports the test models (smoke, the
1212
# esp_nn:: mul, an ESP-NN delegate program and the Cadence-quantized wake
13-
# word), generates and packs the component into a file:// component mirror,
14-
# and makes each model's kernel bundle, all under <dir>.
13+
# word), generates the component with build_idf_component.sh's stamp
14+
# options (default --allow-dev-stamp), packs it into <dir>/dist and a
15+
# file:// component mirror, and makes each model's kernel bundle, all under
16+
# <dir>.
1517
# test_espressif_idf_nightly.sh build <dir> <shard> <shards>
1618
# Needs only <dir> from prepare: installs ESPRESSIF_IDF_VERSION's ESP-IDF
1719
# (default v6.1), lists the idf_component_consumer configurations with
1820
# idf-build-apps (sdkconfig.ci.* times every target, filtered by
1921
# .build-test-rules.yml), builds shard <shard> of <shards> with ESP-IDF's
2022
# own Python environment, and runs each build on its emulator unless the
2123
# rules disable its test. Reports every failure, then fails.
24+
# test_espressif_idf_nightly.sh registry <dir> <registry URL>
25+
# Needs only <dir> from prepare: builds the smoke consumer for ESP32-S3
26+
# with pytorch/executorch of <dir>'s version from the given component
27+
# registry and runs it on the emulator.
2228

2329
set -euo pipefail
2430

2531
et_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
26-
mode=${1:?usage: $0 prepare <dir> | build <dir> <shard> <shards>}
27-
dir=${2:?usage: $0 prepare <dir> | build <dir> <shard> <shards>}
32+
usage="usage: $0 prepare <dir> [stamp options] | build <dir> <shard> <shards> | registry <dir> <URL>"
33+
mode=${1:?${usage}}
34+
dir=${2:?${usage}}
2835
mkdir -p "${dir}"
2936
dir=$(cd "${dir}" && pwd)
3037
cd "${et_root}"
@@ -42,6 +49,12 @@ logged() {
4249
}
4350

4451
prepare() {
52+
local stamp=("${@:3}")
53+
if (( ${#stamp[@]} == 0 )); then
54+
stamp=(--allow-dev-stamp)
55+
fi
56+
# Checks the stamp against git before the model exports.
57+
bash examples/espressif/build_idf_component.sh --version "${stamp[@]}" > /dev/null
4558
export ESPRESSIF_TOOLS_DIR="${ESPRESSIF_TOOLS_DIR:-${RUNNER_TEMP:-/tmp}/executorch-espressif}"
4659
local logs="${dir}/logs" models="${dir}/models"
4760
mkdir -p "${logs}" "${models}"
@@ -71,11 +84,13 @@ print(entry["atol"], entry["rtol"])
7184
rm -r "${models}"/*.bpte "${dir}/espnn"
7285

7386
local component_dir="${dir}/component/executorch"
74-
logged "${logs}/generate.log" bash examples/espressif/build_idf_component.sh \
75-
--allow-dev-stamp --out "${component_dir}"
87+
logged "${logs}/generate.log" bash examples/espressif/build_idf_component.sh "${stamp[@]}" \
88+
--out "${component_dir}"
7689
local version
7790
version=$(sed -n 's/^version: "\(.*\)"$/\1/p' "${component_dir}/idf_component.yml")
78-
(cd "${component_dir}" && compote component pack --name executorch --dest-dir "${dir}/dist")
91+
# The registry links each version to its commit and to the generator.
92+
(cd "${component_dir}" && compote component pack --name executorch --dest-dir "${dir}/dist" \
93+
--commit-sha "$(git -C "${et_root}" rev-parse HEAD)" --repository-path examples/espressif)
7994
python .ci/scripts/esp_idf_mirror.py "${dir}/dist/executorch_${version}.tgz" "${dir}/mirror" \
8095
--namespace pytorch --name executorch
8196
local model
@@ -96,40 +111,15 @@ build() {
96111
mkdir -p "${work}"
97112
work=$(cd "${work}" && pwd)
98113

99-
# ESP-IDF's QEMU needs these libraries, and setup.sh fails its install check
100-
# without them.
101-
if ! python -c '
102-
import ctypes
103-
ctypes.CDLL("libSDL2-2.0.so.0")
104-
ctypes.CDLL("libslirp.so.0")' 2> /dev/null; then
105-
sudo apt-get update
106-
sudo apt-get install -y --no-install-recommends libsdl2-2.0-0 libslirp0
107-
fi
108-
logged "${work}/setup.log" bash examples/espressif/setup.sh --minimal \
109-
--idf-version="${idf_version}" --targets="$(IFS=,; echo "${targets[*]}")"
110-
# shellcheck source=/dev/null
111-
source "${ESPRESSIF_TOOLS_DIR}/setup_path.sh"
112-
113-
# A fresh ESP-IDF Python environment, which cannot import executorch.
114-
local idf_python_env="${work}/idf_python_env"
115-
mkdir -p "${idf_python_env}"
116-
IDF_PYTHON_ENV_PATH="${idf_python_env}" logged "${work}/idf_python_env.log" \
117-
python "${IDF_PATH}/tools/idf_tools.py" install-python-env
118-
"${idf_python_env}/bin/python" -m pip install --quiet "idf-build-apps>=3.1,<4"
119-
export IDF_PYTHON_ENV_PATH="${idf_python_env}"
114+
install_idf "${idf_version}" "$(IFS=,; echo "${targets[*]}")" "${work}"
115+
"${IDF_PYTHON_ENV_PATH}/bin/python" -m pip install --quiet "idf-build-apps>=3.1,<4"
120116
export IDF_COMPONENT_STORAGE_URL="file://${dir}/mirror;default"
121117
export ET_COMPONENT_VERSION ET_CONSUMER_MODELS_DIR="${dir}/models"
122118
ET_COMPONENT_VERSION=$(cat "${dir}/version")
123119

124-
# Built from a copy, so the component manager's state and the consumer's
125-
# generated manifest stay out of the source tree.
126-
local sources="${work}/sources"
127-
rm -rf "${sources}"
128-
mkdir -p "${sources}/examples/espressif/idf_component_smoke"
129-
cp -r examples/espressif/idf_component_consumer "${sources}/examples/espressif/"
130-
cp -r examples/espressif/idf_component_smoke/main "${sources}/examples/espressif/idf_component_smoke/"
131-
local app=examples/espressif/idf_component_consumer
132-
(cd "${sources}" && "${idf_python_env}/bin/idf-build-apps" find -p "${app}" --target all \
120+
local sources="${work}/sources" app=examples/espressif/idf_component_consumer
121+
copy_consumer "${sources}"
122+
(cd "${sources}" && "${IDF_PYTHON_ENV_PATH}/bin/idf-build-apps" find -p "${app}" --target all \
133123
--default-build-targets "${targets[@]}" --config-rules 'sdkconfig.ci.*=' \
134124
--manifest-files "${app}/.build-test-rules.yml" --output-format raw \
135125
--output "${work}/apps.txt")
@@ -155,7 +145,7 @@ PY
155145
while read -r target config test_disabled <&3; do
156146
name="${target}-${config}"
157147
log="${work}/build-${name}.log"
158-
if ! IDF_TARGET="${target}" "${idf_python_env}/bin/python" "${IDF_PATH}/tools/idf.py" \
148+
if ! IDF_TARGET="${target}" "${IDF_PYTHON_ENV_PATH}/bin/python" "${IDF_PATH}/tools/idf.py" \
159149
-C "${project}" -B "${builds}/${name}" -DSDKCONFIG="${builds}/${name}/sdkconfig" \
160150
"-DSDKCONFIG_DEFAULTS=sdkconfig.defaults;sdkconfig.ci.${config}" build \
161151
> "${log}" 2>&1; then
@@ -187,6 +177,122 @@ PY
187177
echo "PASS: shard ${shard}/${shards} of the ESP-IDF ${idf_version} matrix built and ran."
188178
}
189179

180+
# Installs ESP-IDF for the targets and makes a fresh ESP-IDF Python
181+
# environment, which cannot import executorch: release, comma-separated
182+
# targets, work dir.
183+
install_idf() {
184+
local idf_version=$1 targets_csv=$2 work=$3
185+
# ESP-IDF's QEMU needs these libraries, and setup.sh fails its install check
186+
# without them.
187+
if [[ ",${targets_csv}," == *,esp32,* ]] && ! python -c '
188+
import ctypes
189+
ctypes.CDLL("libSDL2-2.0.so.0")
190+
ctypes.CDLL("libslirp.so.0")' 2> /dev/null; then
191+
sudo apt-get update
192+
sudo apt-get install -y --no-install-recommends libsdl2-2.0-0 libslirp0
193+
fi
194+
logged "${work}/setup.log" bash examples/espressif/setup.sh --minimal \
195+
--idf-version="${idf_version}" --targets="${targets_csv}"
196+
# shellcheck source=/dev/null
197+
source "${ESPRESSIF_TOOLS_DIR}/setup_path.sh"
198+
export IDF_PYTHON_ENV_PATH="${work}/idf_python_env"
199+
mkdir -p "${IDF_PYTHON_ENV_PATH}"
200+
logged "${work}/idf_python_env.log" python "${IDF_PATH}/tools/idf_tools.py" install-python-env
201+
}
202+
203+
# Copies the consumer project to a directory, so the component manager's state
204+
# and the consumer's generated manifest stay out of the source tree.
205+
copy_consumer() {
206+
local sources=$1
207+
rm -rf "${sources}"
208+
mkdir -p "${sources}/examples/espressif/idf_component_smoke"
209+
cp -r examples/espressif/idf_component_consumer "${sources}/examples/espressif/"
210+
cp -r examples/espressif/idf_component_smoke/main "${sources}/examples/espressif/idf_component_smoke/"
211+
}
212+
213+
registry() {
214+
local registry_url=${3:?${usage}}
215+
local work="${ESPRESSIF_BUILD_DIR:-${et_root}/cmake-out-esp-idf-registry}"
216+
export ESPRESSIF_TOOLS_DIR="${ESPRESSIF_TOOLS_DIR:-${RUNNER_TEMP:-/tmp}/executorch-espressif}"
217+
mkdir -p "${work}"
218+
work=$(cd "${work}" && pwd)
219+
install_idf v6.1 esp32s3 "${work}"
220+
export ET_COMPONENT_VERSION ET_COMPONENT_REGISTRY_URL="${registry_url}"
221+
export ET_CONSUMER_MODELS_DIR="${dir}/models"
222+
ET_COMPONENT_VERSION=$(cat "${dir}/version")
223+
local storage
224+
if ! storage=$(curl -sSf --retry 3 "${registry_url%/}/api/" | python -c '
225+
import json, sys
226+
print(json.load(sys.stdin)["components_base_url"])'); then
227+
echo "FAIL: ${registry_url}/api/ did not name the registry's storage" >&2
228+
exit 1
229+
fi
230+
# The registry's storage is behind a CDN that caches for five minutes.
231+
local tries=0
232+
until curl -sSf "${storage}/components/pytorch/executorch.json" 2> /dev/null | python -c '
233+
import json, sys
234+
try:
235+
versions = [v["version"] for v in json.load(sys.stdin)["versions"]]
236+
except ValueError:
237+
versions = []
238+
sys.exit(sys.argv[1] not in versions)' "${ET_COMPONENT_VERSION}"; do
239+
if (( ++tries == 30 )); then
240+
echo "FAIL: ${registry_url} did not list pytorch/executorch ${ET_COMPONENT_VERSION}" >&2
241+
exit 1
242+
fi
243+
echo "Waiting for ${registry_url} to list pytorch/executorch ${ET_COMPONENT_VERSION}..."
244+
sleep 30
245+
done
246+
copy_consumer "${work}/sources"
247+
local project="${work}/sources/examples/espressif/idf_component_consumer"
248+
local build="${work}/builds/esp32s3-default" log="${work}/emulator.log"
249+
IDF_TARGET=esp32s3 logged "${work}/build.log" "${IDF_PYTHON_ENV_PATH}/bin/python" \
250+
"${IDF_PATH}/tools/idf.py" -C "${project}" -B "${build}" -DSDKCONFIG="${build}/sdkconfig" \
251+
"-DSDKCONFIG_DEFAULTS=sdkconfig.defaults;sdkconfig.ci.default" build
252+
# The registry must serve the archive that was tested, which --allow-existing
253+
# uploads would not replace. It rewrites the examples' manifests, dropping
254+
# override_path, so those are the only files that may differ.
255+
python - "${dir}/dist/executorch_${ET_COMPONENT_VERSION}.tgz" \
256+
"${project}/managed_components/pytorch__executorch" "${registry_url}" <<'PY'
257+
import hashlib
258+
import sys
259+
import tarfile
260+
from pathlib import Path
261+
262+
263+
def compared(path):
264+
return path not in (".component_hash", "CHECKSUMS.json") and not (
265+
path.startswith("examples/") and path.endswith("/idf_component.yml")
266+
)
267+
268+
269+
with tarfile.open(sys.argv[1]) as archive:
270+
tested = {
271+
Path(m.name).as_posix(): hashlib.sha256(archive.extractfile(m).read()).hexdigest()
272+
for m in archive
273+
if m.isfile()
274+
}
275+
served_dir = Path(sys.argv[2])
276+
served = {
277+
p.relative_to(served_dir).as_posix(): hashlib.sha256(p.read_bytes()).hexdigest()
278+
for p in served_dir.rglob("*")
279+
if p.is_file()
280+
}
281+
differ = sorted(
282+
p for p in tested.keys() | served.keys() if compared(p) and tested.get(p) != served.get(p)
283+
)
284+
if differ:
285+
sys.exit(f"FAIL: {sys.argv[3]} served files unlike the tested archive's: {differ[:10]}")
286+
PY
287+
run_firmware "${project}" "${build}" esp32s3 default "${log}"
288+
if ! grep -qF "smoke: PASS" "${log}" || ! grep -qF "ET_COMPONENT_SMOKE: PASS" "${log}"; then
289+
tail -n 30 "${log}" >&2
290+
echo "FAIL: the consumer from ${registry_url} did not pass on the emulator" >&2
291+
exit 1
292+
fi
293+
echo "PASS: pytorch/executorch ${ET_COMPONENT_VERSION} from ${registry_url} built and ran."
294+
}
295+
190296
# Runs a build on its emulator: project, build dir, target, config, log.
191297
run_firmware() {
192298
local project=$1 build=$2 target=$3 config=$4 log=$5
@@ -217,7 +323,8 @@ run_firmware() {
217323
}
218324

219325
case "${mode}" in
220-
prepare) prepare ;;
326+
prepare) prepare "$@" ;;
221327
build) build "$@" ;;
222-
*) echo "unknown mode ${mode}; use prepare or build" >&2; exit 1 ;;
328+
registry) registry "$@" ;;
329+
*) echo "unknown mode ${mode}; ${usage}" >&2; exit 1 ;;
223330
esac

‎.ci/scripts/upload_espressif_idf_component.sh‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
#!/usr/bin/env bash
2+
# Copyright (c) Meta Platforms, Inc. and affiliates.
3+
# All rights reserved.
4+
#
5+
# This source code is licensed under the BSD-style license found in the
6+
# LICENSE file in the root directory of this source tree.
7+
8+
# Uploads the component archive that test_espressif_idf_nightly.sh prepare made
9+
# to an ESP Component Registry as pytorch/executorch, and succeeds if the
10+
# registry already has that version.
11+
#
12+
# upload_espressif_idf_component.sh <dir> <registry URL> [--dry-run]
13+
#
14+
# Without --dry-run the job needs `id-token: write`: like
15+
# espressif/upload-components-ci-action, this authenticates with a GitHub OIDC
16+
# token for the registry, which accepts it when the component's trusted
17+
# uploader names this repository and the job's environment. --dry-run
18+
# validates the archive and needs no token.
19+
20+
set -euo pipefail
21+
22+
usage="usage: $0 <dir> <registry URL> [--dry-run]"
23+
if (( $# < 2 || $# > 3 )) || [[ $# == 3 && "$3" != --dry-run ]]; then
24+
echo "${usage}" >&2
25+
exit 1
26+
fi
27+
dir=$1 registry=$2 upload=("${@:3}")
28+
version=$(cat "${dir}/version")
29+
30+
python -m pip install --progress-bar off "idf-component-manager==3.1.2"
31+
export IDF_COMPONENT_REGISTRY_URL="${registry}"
32+
if (( ${#upload[@]} == 0 )); then
33+
token=$(curl -sSfG --data-urlencode "audience=${registry}" \
34+
-H "Authorization: Bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN:?needs id-token: write}" \
35+
"${ACTIONS_ID_TOKEN_REQUEST_URL}" | jq -er .value)
36+
echo "::add-mask::${token}"
37+
export IDF_COMPONENT_API_TOKEN="${token}"
38+
fi
39+
compote component upload --namespace pytorch --name executorch \
40+
--archive "${dir}/dist/executorch_${version}.tgz" --allow-existing "${upload[@]}"

‎.github/workflows/nightly-espressif-idf.yml‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ on:
99
- .github/workflows/nightly-espressif-idf.yml
1010
- .ci/scripts/test_espressif_idf_nightly.sh
1111
- .ci/scripts/test_espressif_idf_configure.py
12+
- .ci/scripts/upload_espressif_idf_component.sh
1213
- examples/espressif/idf_component_consumer/**
1314

1415
concurrency:
@@ -59,10 +60,8 @@ jobs:
5960
set -euo pipefail
6061
CONDA_ENV=$(conda env list --json | jq -r '.envs | .[-1]')
6162
conda activate "${CONDA_ENV}"
62-
pip install --progress-bar off "idf-component-manager~=3.1"
63-
version=$(cat "${RUNNER_ARTIFACT_DIR}/version")
64-
compote component upload --namespace pytorch --name executorch \
65-
--archive "${RUNNER_ARTIFACT_DIR}/dist/executorch_${version}.tgz" --dry-run --allow-existing
63+
bash .ci/scripts/upload_espressif_idf_component.sh "${RUNNER_ARTIFACT_DIR}" \
64+
https://components.espressif.com --dry-run
6665
6766
build:
6867
name: ESP-IDF ${{ matrix.idf-version }} (${{ matrix.shard }}/4)

0 commit comments

Comments
 (0)