Skip to content

Commit bd1e01c

Browse files
committed
Test the ESP-IDF component through the component manager on every emulated target
A new CI job builds the component the way a project using the published component gets it. It generates and packs the component, adds the archive to a file:// component mirror (.ci/scripts/esp_idf_mirror.py writes the layout the registry serves), and builds idf_component_consumer, which resolves pytorch/executorch from that mirror at the version just generated and esp-nn from the registry. The builds use a fresh ESP-IDF Python environment that cannot import executorch, so they take the kernel bundles made beforehand. They build a copy of the project, which keeps the component manager's state out of the source tree, and the source and build paths have a space. On ESP-IDF 6.1 the job builds for ESP32 (run on ESP-IDF's QEMU), ESP32-S3, -C3, -C5, -C6 and -P4 (run on esp-emu); on 6.0.1, a second job, for ESP32-S3 and -C6. It also lints the component's manifest, and checks that the lock file records the mirror's archive. Every build runs the smoke model. The ESP32-S3 and -C6 builds also run the MLPerf Tiny streaming wake word model, quantized with the Cadence quantizer and exported by the new export_wakeword_model.py as a bundled program of 40 consecutive frames; all 40 must match the quantized PyTorch model. To run a stateful model, the smoke app now runs all of a bundled program's test cases in order on one loaded method, and split_bundled_program.py can repeat them so single-case models still check a second execute. idf_component_consumer reuses the smoke app's main component, so the two cannot drift apart. setup.sh gains --idf-version and installs QEMU when esp32 is a target. The workflow now also runs for changes to the sources the component vendors (runtime, kernels, schema, extension, codegen, the Cadence backend and build_variables.bzl). Authored with Claude Code.
1 parent a405914 commit bd1e01c

14 files changed

Lines changed: 529 additions & 31 deletions

File tree

‎.ci/scripts/esp_idf_mirror.py‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
"""Add a `compote component pack` archive to a local file:// component mirror.
8+
9+
Writes the layout `compote registry sync` produces:
10+
<mirror>/components/<namespace>/<name>.json
11+
<mirror>/components/<namespace>/<name>/<version>/<archive>, CHECKSUMS.json
12+
so that a project with IDF_COMPONENT_STORAGE_URL=file://<mirror> resolves
13+
<namespace>/<name> exactly as it would from the registry.
14+
"""
15+
16+
import argparse
17+
import json
18+
import shutil
19+
import tarfile
20+
import tempfile
21+
from datetime import datetime, timezone
22+
from pathlib import Path
23+
24+
from idf_component_tools.hash_tools.calculate import hash_dir
25+
from idf_component_tools.hash_tools.checksums import ChecksumsManager
26+
from idf_component_tools.manager import ManifestManager
27+
28+
29+
def main() -> None:
30+
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
31+
parser.add_argument("archive", type=Path)
32+
parser.add_argument("mirror", type=Path)
33+
parser.add_argument("--namespace", required=True)
34+
parser.add_argument("--name", required=True)
35+
args = parser.parse_args()
36+
archive, mirror = args.archive.resolve(), args.mirror.resolve()
37+
38+
with tempfile.TemporaryDirectory() as tmp:
39+
with tarfile.open(archive) as tf:
40+
tf.extractall(tmp, filter="data")
41+
manifest = ManifestManager(tmp, args.name).load()
42+
component_hash = hash_dir(
43+
tmp,
44+
use_gitignore=manifest.use_gitignore,
45+
include=manifest.include_set,
46+
exclude=set(manifest.exclude_set)
47+
| {"**/.component_hash", "**/CHECKSUMS.json"},
48+
exclude_default=False,
49+
)
50+
version = str(manifest.version)
51+
rel_dir = Path("components") / args.namespace / args.name / version
52+
(mirror / rel_dir).mkdir(parents=True, exist_ok=True)
53+
shutil.copy2(archive, mirror / rel_dir / archive.name)
54+
ChecksumsManager(Path(tmp)).dump(mirror / rel_dir)
55+
56+
dependencies = []
57+
for requirement in manifest.requirements:
58+
namespace, _, name = requirement.name.rpartition("/")
59+
is_idf = requirement.source.type == "idf"
60+
dependencies.append(
61+
{
62+
"spec": requirement.version_spec,
63+
"source": requirement.source.type,
64+
"registry_url": None,
65+
"is_public": requirement.is_public,
66+
"require": requirement.is_required,
67+
"name": None if is_idf else name,
68+
"namespace": None if is_idf else namespace,
69+
"rules": [],
70+
"matches": [],
71+
}
72+
)
73+
created_at = datetime.now(timezone.utc).isoformat()
74+
entry = {
75+
"id": "local",
76+
"version": version,
77+
"created_at": created_at,
78+
"description": manifest.description,
79+
"component_hash": component_hash,
80+
"url": (rel_dir / archive.name).as_posix(),
81+
"targets": list(manifest.targets or []),
82+
"maintainers": [],
83+
"docs": {},
84+
"dependencies": dependencies,
85+
"license": None,
86+
"examples": [],
87+
"tags": [],
88+
"checksums": (rel_dir / "CHECKSUMS.json").as_posix(),
89+
"yanked_at": None,
90+
"yanked_message": None,
91+
}
92+
index = mirror / "components" / args.namespace / f"{args.name}.json"
93+
data = (
94+
json.loads(index.read_text())
95+
if index.exists()
96+
else {
97+
"name": args.name,
98+
"namespace": args.namespace,
99+
"created_at": created_at,
100+
"featured": False,
101+
"versions": [],
102+
}
103+
)
104+
data["versions"] = [v for v in data["versions"] if v["version"] != version]
105+
data["versions"].append(entry)
106+
index.write_text(json.dumps(data))
107+
print(f"{args.namespace}/{args.name} {version} -> {index}")
108+
109+
110+
if __name__ == "__main__":
111+
main()

‎.ci/scripts/test_espressif_idf_component.sh‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,15 +50,16 @@ python examples/espressif/export_espnn_test_programs.py --output-dir "${build_di
5050
--only conv-act-pool-linear 2>&1 | tee -a "${build_dir}/export.log"
5151
cp "${build_dir}/espnn/conv-act-pool-linear.bpte" "${build_dir}/delegate.bpte"
5252
split=examples/espressif/idf_component_smoke/split_bundled_program.py
53-
python "${split}" "${build_dir}/smoke.bpte" --atol 0 --rtol 0
53+
# Each case twice, so a second execute of the same method is checked too.
54+
python "${split}" "${build_dir}/smoke.bpte" --atol 0 --rtol 0 --repeat 2
5455
# The tolerance test_mul.py uses.
55-
python "${split}" "${build_dir}/mul.bpte" --atol 0 --rtol 1.2e-7
56+
python "${split}" "${build_dir}/mul.bpte" --atol 0 --rtol 1.2e-7 --repeat 2
5657
read -r atol rtol < <(python -c '
5758
import json, sys
5859
(entry,) = json.load(open(sys.argv[1]))
5960
print(entry["atol"], entry["rtol"])
6061
' "${build_dir}/espnn/manifest.json")
61-
python "${split}" "${build_dir}/delegate.bpte" --atol "${atol}" --rtol "${rtol}"
62+
python "${split}" "${build_dir}/delegate.bpte" --atol "${atol}" --rtol "${rtol}" --repeat 2
6263
cpu_models="${build_dir}/smoke.pte;${build_dir}/mul.pte"
6364
models="${cpu_models};${build_dir}/delegate.pte"
6465

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
#!/usr/bin/env bash
2+
# Copyright (c) Meta Platforms, Inc. and affiliates.
3+
# All rights reserved.
4+
#
5+
# This source code is licensed under the BSD-style license found in the
6+
# LICENSE file in the root directory of this source tree.
7+
8+
# Builds the ExecuTorch ESP-IDF component the way a project using the published
9+
# component gets it: generates and packs the component, adds the archive to a
10+
# file:// component mirror, and builds the idf_component_consumer project, which
11+
# resolves executorch from that mirror, with ESP-IDF's own Python environment
12+
# (no ExecuTorch) and with source and build paths that have a space. Each build
13+
# runs the smoke model on the emulator; the ESP32-S3 and ESP32-C6 builds also
14+
# run a Cadence-quantized streaming wake word model frame by frame against
15+
# PyTorch.
16+
# ESPRESSIF_IDF_VERSION selects the ESP-IDF release: v6.1 builds every target
17+
# an emulator runs (QEMU for ESP32, esp-emu for the rest), other releases
18+
# ESP32-S3 and ESP32-C6. Also lints the component's manifest.
19+
20+
set -euo pipefail
21+
22+
et_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
23+
idf_version="${ESPRESSIF_IDF_VERSION:-v6.1}"
24+
build_dir="${ESPRESSIF_BUILD_DIR:-${et_root}/cmake-out-esp-idf-consumer-${idf_version}}"
25+
export ESPRESSIF_TOOLS_DIR="${ESPRESSIF_TOOLS_DIR:-${RUNNER_TEMP:-/tmp}/executorch-espressif-${idf_version}}"
26+
mkdir -p "${build_dir}"
27+
build_dir=$(cd "${build_dir}" && pwd)
28+
cd "${et_root}"
29+
30+
if [[ "${idf_version}" == v6.1 ]]; then
31+
targets=(esp32s3 esp32 esp32c3 esp32c5 esp32c6 esp32p4)
32+
else
33+
targets=(esp32s3 esp32c6)
34+
fi
35+
36+
collect_artifacts() {
37+
if [[ -n "${RUNNER_ARTIFACT_DIR:-}" ]]; then
38+
mkdir -p "${RUNNER_ARTIFACT_DIR}/esp-idf-consumer"
39+
cp "${build_dir}"/*.log "${RUNNER_ARTIFACT_DIR}/esp-idf-consumer/" || true
40+
fi
41+
}
42+
trap collect_artifacts EXIT
43+
44+
# ESP-IDF's QEMU needs these libraries, and setup.sh fails its install check
45+
# without them.
46+
if [[ " ${targets[*]} " == *" esp32 "* ]] && ! python -c '
47+
import ctypes
48+
ctypes.CDLL("libSDL2-2.0.so.0")
49+
ctypes.CDLL("libslirp.so.0")' 2> /dev/null; then
50+
sudo apt-get update
51+
sudo apt-get install -y --no-install-recommends libsdl2-2.0-0 libslirp0
52+
fi
53+
bash examples/espressif/setup.sh --minimal --idf-version="${idf_version}" \
54+
--targets="$(IFS=,; echo "${targets[*]}")" 2>&1 | tee "${build_dir}/setup.log"
55+
# shellcheck source=/dev/null
56+
source "${ESPRESSIF_TOOLS_DIR}/setup_path.sh"
57+
58+
python examples/espressif/export_smoke_model.py --output "${build_dir}/smoke.bpte" \
59+
2>&1 | tee "${build_dir}/export.log"
60+
python examples/espressif/export_wakeword_model.py --output "${build_dir}/wakeword.bpte" \
61+
2>&1 | tee -a "${build_dir}/export.log"
62+
split=examples/espressif/idf_component_smoke/split_bundled_program.py
63+
python "${split}" "${build_dir}/smoke.bpte" --atol 0 --rtol 0 --repeat 2
64+
python "${split}" "${build_dir}/wakeword.bpte" --atol 1e-6 --rtol 0
65+
66+
component_dir="${build_dir}/component/executorch"
67+
bash examples/espressif/build_idf_component.sh --out "${component_dir}" \
68+
2>&1 | tee "${build_dir}/generate.log"
69+
compote manifest lint "${component_dir}/idf_component.yml"
70+
version=$(sed -n 's/^version: "\(.*\)"$/\1/p' "${component_dir}/idf_component.yml")
71+
(cd "${component_dir}" && compote component pack --name executorch --dest-dir "${build_dir}/dist")
72+
archive="${build_dir}/dist/executorch_${version}.tgz"
73+
mirror="${build_dir}/mirror"
74+
rm -rf "${mirror}"
75+
python .ci/scripts/esp_idf_mirror.py "${archive}" "${mirror}" \
76+
--namespace pytorch --name executorch
77+
78+
kernels() {
79+
local out=$1
80+
shift
81+
python "${component_dir}/tools/gen_kernels.py" "$@" --out "${out}" \
82+
2>&1 | tee -a "${build_dir}/gen_kernels.log"
83+
}
84+
kernels "${build_dir}/smoke_kernels" --pte "${build_dir}/smoke.pte"
85+
kernels "${build_dir}/wakeword_kernels" --pte "${build_dir}/smoke.pte" \
86+
--pte "${build_dir}/wakeword.pte"
87+
88+
# A fresh ESP-IDF Python environment, which cannot import executorch.
89+
idf_python_env="${build_dir}/idf_python_env"
90+
mkdir -p "${idf_python_env}"
91+
IDF_PYTHON_ENV_PATH="${idf_python_env}" python "${IDF_PATH}/tools/idf_tools.py" \
92+
install-python-env 2>&1 | tee "${build_dir}/idf_python_env.log"
93+
# Built from a copy, so the component manager's state and the consumer's
94+
# generated manifest stay out of the source tree, under a path with a space.
95+
sources="${build_dir}/consumer sources"
96+
rm -rf "${sources}"
97+
mkdir -p "${sources}/idf_component_smoke"
98+
cp -r examples/espressif/idf_component_consumer "${sources}/"
99+
cp -r examples/espressif/idf_component_smoke/main \
100+
examples/espressif/idf_component_smoke/sdkconfig.defaults "${sources}/idf_component_smoke/"
101+
project="${sources}/idf_component_consumer"
102+
firmware_root="${build_dir}/consumer builds"
103+
104+
# Runs idf.py on the consumer with ESP-IDF's Python and the mirror: build name,
105+
# target, then idf.py arguments.
106+
consumer() {
107+
local name=$1 target=$2
108+
shift 2
109+
IDF_TARGET="${target}" IDF_PYTHON_ENV_PATH="${idf_python_env}" \
110+
IDF_COMPONENT_STORAGE_URL="file://${mirror};default" \
111+
"${idf_python_env}/bin/python" "${IDF_PATH}/tools/idf.py" -C "${project}" \
112+
-B "${firmware_root}/${name}" -DSDKCONFIG="${firmware_root}/${name}/sdkconfig" \
113+
-DET_COMPONENT_VERSION="${version}" "$@"
114+
}
115+
116+
# Runs a build on its emulator and checks its output: build name, target, then
117+
# the per-model markers.
118+
run_firmware() {
119+
local name=$1 target=$2
120+
shift 2
121+
local firmware_dir="${firmware_root}/${name}" log="${build_dir}/emulator-${name}.log"
122+
if [[ "${target}" == esp32 ]]; then
123+
# QEMU has no exit-on string, so stop it once the app is done.
124+
setsid bash -c 'consumer "$@" qemu' _ "${name}" "${target}" < /dev/null > "${log}" 2>&1 &
125+
local pid=$!
126+
for _ in $(seq 240); do
127+
if grep -qF "Program complete." "${log}"; then
128+
break
129+
fi
130+
sleep 0.5
131+
done
132+
kill -- "-${pid}" 2> /dev/null || true
133+
wait "${pid}" || true
134+
else
135+
consumer "${name}" "${target}" merge-bin > "${build_dir}/merge-${name}.log" 2>&1
136+
timeout 130s esp-emu --chip "${target}" \
137+
--firmware "${firmware_dir}/merged-binary.bin" \
138+
--elf "${firmware_dir}/et_component_smoke.elf" \
139+
--net user --timeout 120s --exit-on 'Program complete.' \
140+
> "${log}" 2>&1 || true
141+
fi
142+
for marker in "$@" "ET_COMPONENT_SMOKE: PASS" "Program complete."; do
143+
if ! grep -qF "${marker}" "${log}"; then
144+
echo "FAIL: ${name} printed no '${marker}'; see emulator-${name}.log" >&2
145+
exit 1
146+
fi
147+
done
148+
echo "OK: ${name} passed on the ${target} emulator"
149+
}
150+
export -f consumer
151+
export idf_python_env mirror project firmware_root version
152+
153+
for target in "${targets[@]}"; do
154+
if [[ "${target}" == esp32s3 || "${target}" == esp32c6 ]]; then
155+
consumer "${target}" "${target}" \
156+
-DET_SMOKE_MODELS="${build_dir}/smoke.pte;${build_dir}/wakeword.pte" \
157+
-DET_SMOKE_KERNELS="${build_dir}/wakeword_kernels" build \
158+
> "${build_dir}/firmware-${target}.log" 2>&1
159+
run_firmware "${target}" "${target}" "smoke: PASS" "wakeword: PASS, 40/40 cases"
160+
else
161+
consumer "${target}" "${target}" -DET_SMOKE_MODELS="${build_dir}/smoke.pte" \
162+
-DET_SMOKE_KERNELS="${build_dir}/smoke_kernels" build \
163+
> "${build_dir}/firmware-${target}.log" 2>&1
164+
run_firmware "${target}" "${target}" "smoke: PASS"
165+
fi
166+
python - "${project}/dependencies.lock" "${mirror}/components/pytorch/executorch.json" <<'PY'
167+
import json
168+
import sys
169+
170+
import yaml
171+
172+
entry = yaml.safe_load(open(sys.argv[1]))["dependencies"]["pytorch/executorch"]
173+
(mirrored,) = json.load(open(sys.argv[2]))["versions"]
174+
if (str(entry["version"]), entry.get("component_hash")) != (
175+
mirrored["version"],
176+
mirrored["component_hash"],
177+
):
178+
sys.exit(f"FAIL: pytorch/executorch did not come from the mirror: {entry}")
179+
PY
180+
done
181+
echo "PASS: the packed ESP-IDF component built from a mirror and ran on every target."

‎.github/workflows/test-espressif.yml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,23 @@ on:
99
paths:
1010
- backends/espressif/**
1111
- examples/espressif/**
12+
# Vendored into the ESP-IDF component.
13+
- runtime/**
14+
- kernels/**
15+
- schema/**
16+
- extension/**
17+
- codegen/**
18+
- backends/cadence/**
19+
- shim_et/xplat/executorch/build/build_variables.bzl
20+
# Exported by the consumer test.
21+
- examples/models/mlperf_tiny/**
22+
- devtools/bundled_program/**
1223
- tools/cmake/preset/esp_baremetal.cmake
1324
- CMakePresets.json
1425
- .ci/scripts/test_espressif.sh
1526
- .ci/scripts/test_espressif_idf_component.sh
27+
- .ci/scripts/test_espressif_idf_consumer.sh
28+
- .ci/scripts/esp_idf_mirror.py
1629
- .github/workflows/test-espressif.yml
1730
schedule:
1831
- cron: '0 4 * * 1'
@@ -74,3 +87,29 @@ jobs:
7487
source .ci/scripts/utils.sh
7588
install_executorch --use-pt-pinned-commit --minimal
7689
bash .ci/scripts/test_espressif_idf_component.sh
90+
91+
test-idf-consumer:
92+
name: ESP-IDF component consumer (${{ matrix.idf-version }})
93+
needs: docker-image
94+
strategy:
95+
fail-fast: false
96+
matrix:
97+
idf-version: [v6.1, v6.0.1]
98+
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
99+
permissions:
100+
id-token: write
101+
contents: read
102+
with:
103+
runner: mt-l-x86iavx512-8-64
104+
docker-image: ${{ needs.docker-image.outputs.docker-registry }}/ci-image:executorch-ubuntu-22.04-clang12-${{ needs.docker-image.outputs.ci-docker-hash }}
105+
submodules: recursive
106+
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
107+
timeout: 45
108+
upload-artifact: test-report-esp-idf-consumer-${{ matrix.idf-version }}
109+
script: |
110+
set -euo pipefail
111+
CONDA_ENV=$(conda env list --json | jq -r '.envs | .[-1]')
112+
conda activate "${CONDA_ENV}"
113+
source .ci/scripts/utils.sh
114+
install_executorch --use-pt-pinned-commit --minimal
115+
ESPRESSIF_IDF_VERSION="${{ matrix.idf-version }}" bash .ci/scripts/test_espressif_idf_consumer.sh

‎examples/espressif/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,4 @@ sdkconfig
66
sdkconfig.old
77
__pycache__/
88
camera_stream/models/
9+
idf_component_consumer/main/idf_component.yml

‎examples/espressif/README.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,11 @@ application.
3535
Consumers need ESP-IDF, plus the `executorch` Python package of the
3636
same release to make kernel bundles — no host CMake build, no flatc.
3737
`idf_component_smoke/` is a minimal consumer that CI generates, builds
38-
and runs on the ESP32-S3 emulator
39-
(`.ci/scripts/test_espressif_idf_component.sh`).
38+
and runs on the emulator (`.ci/scripts/test_espressif_idf_component.sh`).
39+
`idf_component_consumer/` builds the same app from the packed component
40+
through the component manager, with ESP-IDF's own Python, for every
41+
target an emulator runs, on ESP-IDF 6.1 and 6.0.1
42+
(`.ci/scripts/test_espressif_idf_consumer.sh`).
4043

4144
The executor runner below remains the reference for the prebuilt-library
4245
flow and for benchmarking.
@@ -262,6 +265,9 @@ prints the corresponding activation command. It also accepts `IDF_PATH` and
262265
`IDF_TOOLS_PATH` for an existing ESP-IDF installation. The setup script checks
263266
the SDK version before installing dependencies into the active environment.
264267
Use `--skip-idf` or `--skip-emulator` to install only the tools you need.
268+
`--targets=<chip>,...` installs the compilers for other targets (and, for
269+
`esp32`, ESP-IDF's QEMU), and `--idf-version=v6.0.1` installs that ESP-IDF
270+
release instead of 6.1; use a separate `ESPRESSIF_TOOLS_DIR` for it.
265271

266272
### 2. Export the test model
267273

0 commit comments

Comments
 (0)