Skip to content

Commit db7697c

Browse files
rascanirascani
committed
Add a nightly matrix for the ESP-IDF component
nightly-espressif-idf.yml runs the component across every target, ESP-IDF release and configuration that the per-PR jobs only sample. A prepare job generates and packs the component into a file:// mirror and exports the smoke, esp_nn:: mul, ESP-NN delegate and Cadence wake word models with their kernel bundles. Twelve build jobs (ESP-IDF 6.0.1, 6.1 and master, four shards each; master may fail without failing the run) take that artifact and nothing else: they list the idf_component_consumer configurations with idf-build-apps (each sdkconfig.ci.* file for ESP32, -S3, -C3, -C5, -C6 and -P4, filtered by .build-test-rules.yml), build their shard from the mirror with ESP-IDF's own Python environment, and run each build on QEMU or esp-emu. The configurations are default, nolog (logging compiled out), os (-Os), wakeword, delegate and oplib, plus a build-only ESP32-P4 configuration for silicon before rev3, which esp-emu does not run. idf-build-apps passes no -D arguments, so the consumer now reads ET_COMPONENT_VERSION from the environment and, without ET_SMOKE_MODELS, runs CONFIG_ET_CONSUMER_MODEL's .pte and bundle from ET_CONSUMER_MODELS_DIR. It has its own sdkconfig.defaults instead of overriding SDKCONFIG_DEFAULTS, which idf-build-apps sets. The script builds each listed configuration itself with its own sdkconfig, because idf-build-apps keeps one sdkconfig in the project directory, which the next build overwrites before the emulator run needs it. test_espressif_idf_component.sh --all-ops registers every op of every kernel library (296 today) in one ESP32-S3 and one ESP32-C6 image with a 7 MB app partition, so every vendored kernel must compile under ESP-IDF's flags and every registration must resolve when the image links; both images then run the smoke model. Jobs on macOS and Windows configure the two-model consumer with the prepared component, once with its bundle and once in automatic mode, and check that both select the same kernels. The registry validation (compote component upload --dry-run --allow-existing) runs here on the packed archive. setup.sh accepts an ESP-IDF branch such as master. Locally the ESP-IDF 6.1 matrix (37 builds) passed on the emulators, as did --all-ops and the configure script on Linux. The workflow also runs on pull requests that change it, since workflow_dispatch only works from the default branch. Authored with Claude Code.
1 parent 2381b16 commit db7697c

19 files changed

Lines changed: 638 additions & 33 deletions

‎.ci/scripts/test_espressif_idf_component.sh‎

Lines changed: 57 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,16 @@
1515
# component's own example runs too. Also checks the version stamping, the
1616
# manifest and SBOM, that stale or mismatched bundles fail at configure time,
1717
# and that an app declaring nothing links to the same size as one without the
18-
# component.
18+
# component. With --all-ops (nightly), it instead builds and runs one ESP32-S3
19+
# and one ESP32-C6 image that register every op of every kernel library.
1920

2021
set -euo pipefail
2122

23+
all_ops=false
24+
if [[ "${1:-}" == --all-ops ]]; then
25+
all_ops=true
26+
fi
27+
2228
et_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
2329
build_dir="${ESPRESSIF_BUILD_DIR:-${et_root}/cmake-out-esp-idf-component}"
2430
export ESPRESSIF_TOOLS_DIR="${ESPRESSIF_TOOLS_DIR:-${RUNNER_TEMP:-/tmp}/executorch-espressif}"
@@ -128,6 +134,56 @@ expect_configure_error() {
128134
echo "OK: ${name} failed at configure with '${expected}'"
129135
}
130136

137+
# Runs a build on the emulator and checks its output: build name, chip, then
138+
# the per-model markers.
139+
run_firmware() {
140+
local name=$1 chip=$2
141+
shift 2
142+
local firmware_dir="${build_dir}/firmware-${name}"
143+
IDF_TARGET="${chip}" idf.py -C "${project}" -B "${firmware_dir}" merge-bin \
144+
> "${build_dir}/merge-${name}.log" 2>&1
145+
local elf=("${firmware_dir}"/*.elf)
146+
timeout 130s esp-emu --chip "${chip}" \
147+
--firmware "${firmware_dir}/merged-binary.bin" --elf "${elf[0]}" \
148+
--net user --timeout 120s --exit-on 'Program complete.' \
149+
2>&1 | tee "${build_dir}/emulator-${name}.log"
150+
for marker in "$@" "ET_COMPONENT_SMOKE: PASS" "Program complete."; do
151+
if ! grep -qF "${marker}" "${build_dir}/emulator-${name}.log"; then
152+
echo "FAIL: ${name} printed no '${marker}'; see emulator-${name}.log" >&2
153+
exit 1
154+
fi
155+
done
156+
echo "OK: ${name} passed on the ${chip} emulator"
157+
}
158+
159+
# --all-ops: one image per architecture registering every op of every kernel
160+
# library, so every vendored kernel must compile under ESP-IDF's flags and every
161+
# registration must resolve when the image links. Skips the other checks.
162+
if [[ "${all_ops}" == true ]]; then
163+
mkdir -p "${build_dir}/all_ops"
164+
oplists=()
165+
for yaml in "${component_dir}"/tools/libraries/*.yaml; do
166+
oplist="${build_dir}/all_ops/$(basename "${yaml}")"
167+
python -m executorch.codegen.tools.gen_oplist --output_path="${oplist}" \
168+
--ops_schema_yaml_path="${yaml}" > /dev/null
169+
oplists+=(--oplist "${oplist}")
170+
done
171+
gen_kernels "${build_dir}/all_ops_kernels" --pte "${build_dir}/smoke.pte" "${oplists[@]}"
172+
printf 'nvs,data,nvs,0x9000,0x6000\nphy_init,data,phy,0xf000,0x1000\nfactory,app,factory,0x10000,7M\n' \
173+
> "${build_dir}/all_ops/partitions.csv"
174+
printf 'CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y\nCONFIG_PARTITION_TABLE_CUSTOM=y\nCONFIG_PARTITION_TABLE_CUSTOM_FILENAME="%s"\n' \
175+
"${build_dir}/all_ops/partitions.csv" > "${build_dir}/all_ops/sdkconfig.defaults"
176+
for target in esp32s3 esp32c6; do
177+
consumer "all-ops-${target}" "${target}" -DET_COMPONENT_DIR="${component_dir}" \
178+
"-DSDKCONFIG_DEFAULTS=sdkconfig.defaults;${build_dir}/all_ops/sdkconfig.defaults" \
179+
-DET_SMOKE_MODELS="${build_dir}/smoke.pte" \
180+
-DET_SMOKE_KERNELS="${build_dir}/all_ops_kernels" build
181+
run_firmware "all-ops-${target}" "${target}" "smoke: PASS"
182+
done
183+
echo "PASS: every op of every kernel library linked and booted on ESP32-S3 and ESP32-C6."
184+
exit 0
185+
fi
186+
131187
# A re-exported smoke.pte.
132188
mkdir -p "${build_dir}/stale"
133189
cp "${build_dir}/smoke.pte" "${build_dir}/stale/smoke.pte"
@@ -201,28 +257,6 @@ if ! tr -s ' \n' ' ' < "${build_dir}/firmware-nano.log" \
201257
exit 1
202258
fi
203259

204-
# Runs a build on the emulator and checks its output: build name, chip, then
205-
# the per-model markers.
206-
run_firmware() {
207-
local name=$1 chip=$2
208-
shift 2
209-
local firmware_dir="${build_dir}/firmware-${name}"
210-
IDF_TARGET="${chip}" idf.py -C "${project}" -B "${firmware_dir}" merge-bin \
211-
> "${build_dir}/merge-${name}.log" 2>&1
212-
local elf=("${firmware_dir}"/*.elf)
213-
timeout 130s esp-emu --chip "${chip}" \
214-
--firmware "${firmware_dir}/merged-binary.bin" --elf "${elf[0]}" \
215-
--net user --timeout 120s --exit-on 'Program complete.' \
216-
2>&1 | tee "${build_dir}/emulator-${name}.log"
217-
for marker in "$@" "ET_COMPONENT_SMOKE: PASS" "Program complete."; do
218-
if ! grep -qF "${marker}" "${build_dir}/emulator-${name}.log"; then
219-
echo "FAIL: ${name} printed no '${marker}'; see emulator-${name}.log" >&2
220-
exit 1
221-
fi
222-
done
223-
echo "OK: ${name} passed on the ${chip} emulator"
224-
}
225-
226260
# No bundle declared: the build makes one with its own Python, which has
227261
# executorch here, and it must match the one gen_kernels.py made above.
228262
consumer all-esp32s3 esp32s3 -DET_COMPONENT_DIR="${component_dir}" \

‎.ci/scripts/test_espressif_idf_configure.py‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
#!/usr/bin/env python3
2+
# Copyright (c) Meta Platforms, Inc. and affiliates.
3+
# All rights reserved.
4+
#
5+
# This source code is licensed under the BSD-style license found in the
6+
# LICENSE file in the root directory of this source tree.
7+
8+
"""Configure the ESP-IDF component's two-model consumer on this host.
9+
10+
For the nightly macOS and Windows jobs. Installs ESP-IDF and its ESP32-S3 tools
11+
under --work-dir with ESP-IDF's own Python environment, then runs
12+
`idf.py reconfigure` on idf_component_smoke with the smoke and mul models and
13+
the component from test_espressif_idf_nightly.sh prepare: once with the kernel
14+
bundle prepare made, and once in automatic mode, where this Python (which must
15+
import executorch) makes the bundle. Both bundles must select the same kernels.
16+
"""
17+
18+
import argparse
19+
import json
20+
import os
21+
import subprocess
22+
import sys
23+
from pathlib import Path
24+
25+
ET_ROOT = Path(__file__).resolve().parents[2]
26+
27+
28+
def run(cmd, env):
29+
print("+", " ".join(str(c) for c in cmd), flush=True)
30+
subprocess.run([str(c) for c in cmd], env=env, check=True)
31+
32+
33+
def idf_environment(work, idf_version):
34+
idf_path = work / "esp-idf"
35+
env = dict(
36+
os.environ,
37+
IDF_PATH=str(idf_path),
38+
IDF_TOOLS_PATH=str(work / "idf-tools"),
39+
IDF_PYTHON_ENV_PATH=str(work / "idf_python_env"),
40+
)
41+
# windows_job runs under Git Bash, whose MSYSTEM makes idf_tools.py refuse to run.
42+
env.pop("MSYSTEM", None)
43+
if not idf_path.exists():
44+
run(
45+
["git", "-c", "core.longpaths=true", "clone", "--depth", "1"]
46+
+ ["--branch", idf_version, "--recurse-submodules", "--shallow-submodules"]
47+
+ ["https://github.com/espressif/esp-idf.git", idf_path],
48+
env,
49+
)
50+
idf_tools = [sys.executable, idf_path / "tools/idf_tools.py"]
51+
run(
52+
idf_tools + ["install", "--targets", "esp32s3", "required", "cmake", "ninja"],
53+
env,
54+
)
55+
# install-python-env fails if IDF_PYTHON_ENV_PATH doesn't exist yet.
56+
Path(env["IDF_PYTHON_ENV_PATH"]).mkdir(exist_ok=True)
57+
run(idf_tools + ["install-python-env"], env)
58+
exported = subprocess.run(
59+
[str(c) for c in idf_tools + ["export", "--format", "key-value"]],
60+
env=env,
61+
check=True,
62+
capture_output=True,
63+
text=True,
64+
).stdout
65+
for line in exported.splitlines():
66+
key, _, value = line.partition("=")
67+
if key == "PATH":
68+
value = value.replace("%PATH%", env["PATH"]).replace("$PATH", env["PATH"])
69+
if key:
70+
env[key] = value
71+
scripts = "Scripts/python.exe" if os.name == "nt" else "bin/python"
72+
return env, Path(env["IDF_PYTHON_ENV_PATH"]) / scripts
73+
74+
75+
def main(argv):
76+
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
77+
parser.add_argument(
78+
"--inputs", type=Path, required=True, help="prepare's directory"
79+
)
80+
parser.add_argument("--work-dir", type=Path, required=True)
81+
parser.add_argument("--idf-version", default="v6.1")
82+
args = parser.parse_args(argv)
83+
work = args.work_dir.resolve()
84+
work.mkdir(parents=True, exist_ok=True)
85+
inputs = args.inputs.resolve()
86+
env, idf_python = idf_environment(work, args.idf_version)
87+
env["IDF_TARGET"] = "esp32s3"
88+
89+
models = ";".join(
90+
(inputs / "models" / f"{m}.pte").as_posix() for m in ("smoke", "mul")
91+
)
92+
common = [
93+
idf_python,
94+
Path(env["IDF_PATH"]) / "tools/idf.py",
95+
"-C",
96+
ET_ROOT / "examples/espressif/idf_component_smoke",
97+
f"-DET_COMPONENT_DIR={(inputs / 'component/executorch').as_posix()}",
98+
f"-DET_SMOKE_MODELS={models}",
99+
]
100+
bundled = work / "bundled"
101+
run(
102+
common
103+
+ ["-B", bundled, f"-DSDKCONFIG={(bundled / 'sdkconfig').as_posix()}"]
104+
+ [f"-DET_SMOKE_KERNELS={(inputs / 'models/smoke_mul_kernels').as_posix()}"]
105+
+ ["reconfigure"],
106+
env,
107+
)
108+
automatic = work / "automatic"
109+
run(
110+
common
111+
+ ["-B", automatic, f"-DSDKCONFIG={(automatic / 'sdkconfig').as_posix()}"]
112+
+ [f"-DEXECUTORCH_PYTHON={Path(sys.executable).as_posix()}", "reconfigure"],
113+
env,
114+
)
115+
116+
def selection(bundle):
117+
kernels = json.loads((bundle / "kernels.json").read_text())
118+
return {k: kernels[k] for k in ("models", "libraries", "prim_ops")}
119+
120+
made = selection(automatic / "executorch_kernels")
121+
if made != selection(inputs / "models/smoke_mul_kernels"):
122+
sys.exit(f"FAIL: automatic mode selected {made}, unlike prepare's bundle")
123+
print(
124+
"PASS: the two-model consumer configured with a bundle and in automatic mode."
125+
)
126+
127+
128+
if __name__ == "__main__":
129+
main(sys.argv[1:])

‎.ci/scripts/test_espressif_idf_consumer.sh‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -96,8 +96,7 @@ sources="${build_dir}/consumer sources"
9696
rm -rf "${sources}"
9797
mkdir -p "${sources}/idf_component_smoke"
9898
cp -r examples/espressif/idf_component_consumer "${sources}/"
99-
cp -r examples/espressif/idf_component_smoke/main \
100-
examples/espressif/idf_component_smoke/sdkconfig.defaults "${sources}/idf_component_smoke/"
99+
cp -r examples/espressif/idf_component_smoke/main "${sources}/idf_component_smoke/"
101100
project="${sources}/idf_component_consumer"
102101
firmware_root="${build_dir}/consumer builds"
103102

0 commit comments

Comments
 (0)