Add a nightly matrix for the ESP-IDF component (#24711)

nightly-espressif-idf.yml runs the component across every target,
ESP-IDF release and configuration that the per-PR jobs only sample. A
prepare job generates and packs the component and exports the smoke,
esp_nn:: mul, ESP-NN delegate and Cadence wake word models with their
kernel bundles, plus an all_ops bundle that registers every op of every
kernel library. Twelve build jobs (ESP-IDF 6.0.1, 6.1 and master, four
shards each) take that artifact and nothing else: they serve the archive
from a file:// mirror, list the idf_component_consumer configurations
with idf-build-apps (each sdkconfig.ci.* file for ESP32, -S3, -C3, -C5,
-C6 and -P4, filtered by .build-test-rules.yml), build their shard from
the mirror with ESP-IDF's own Python environment, check that every
setting of its sdkconfig.ci.* file reached the build's sdkconfig, since
ESP-IDF only warns about an unknown symbol, and run each build on QEMU
or esp-emu. Each job records the ESP-IDF commit it tested and uploads
its build and emulator logs to S3, where they stay 14 days instead of
GitHub's 2. The configurations are default (the smoke model alone);
nolog (logging compiled out), perf (ExecuTorch built without -Os, and
the project for speed) and debug (debug assertions, and full program
verification, which the app then runs on each model as it loads it),
which each run all four models, so the delegate and the quantized,
Cadence and esp_nn:: kernels run under each setting; wakeword, delegate
and oplib, each a model with its own bundle, on ESP32-S3 and -C6, since
the four-model configurations already run those models everywhere;
all_ops, also on ESP32-S3 and -C6, with an 8 MB flash layout, so every
vendored kernel compiles under ESP-IDF's flags and every registration
resolves when the image links; plus a build-only ESP32-P4 configuration
for silicon before rev3, which esp-emu does not run.

The consumer takes the model directory as ET_CONSUMER_MODELS_DIR, from
which CONFIG_ET_CONSUMER_MODEL picks a list of models and their combined
bundle. It has its own sdkconfig.defaults instead of overriding
SDKCONFIG_DEFAULTS. The script builds each listed configuration itself
with its own sdkconfig, because idf-build-apps keeps one sdkconfig in
the project directory, which the next build overwrites before the
emulator run needs it, and runs idf-build-apps from a venv of its own so
ESP-IDF's environment stays as ESP-IDF installs it.

Jobs on macOS and Windows configure the two-model idf_component_smoke
project with the packed component, once with its bundle and once in
automatic mode, and check that both select the same kernels. The prepare
job also runs the registry's validation (compote component upload
--dry-run --allow-existing) on the packed archive, and the build jobs
check that ESP-IDF's Python environment cannot import executorch.
setup.sh accepts an ESP-IDF branch such as master. The workflow also
runs for the ciflow/nightly label, and on pull requests that change the
matrix itself (the workflow, its two scripts and
idf_component_consumer/), since workflow_dispatch only works from the
default branch and test-espressif.yml already builds and runs the packed
component on every pull request that can change it. Pull requests skip
ESP-IDF master, which breaks on its own schedule.

Authored with Claude Code.

Author: rascani

.ci/scripts/test_espressif_idf_configure.py

@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Configure idf_component_smoke with two models on this host.
+
+For the nightly macOS and Windows jobs. Installs ESP-IDF and its ESP32-S3 tools
+under --work-dir/<--idf-version> with ESP-IDF's own Python environment, then runs
+`idf.py reconfigure` on idf_component_smoke with the smoke and mul models and
+the component packed by test_espressif_idf_nightly.sh prepare: once with the
+kernel bundle prepare made, and once in automatic mode, where this Python
+(which must import executorch) makes the bundle. Both bundles must select the
+same models, kernels, prim ops and backends.
+"""
+
+import argparse
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tarfile
+from pathlib import Path
+
+ET_ROOT = Path(__file__).resolve().parents[2]
+
+
+def run(cmd, env):
+    print("+", " ".join(str(c) for c in cmd), flush=True)
+    subprocess.run([str(c) for c in cmd], env=env, check=True)
+
+
+def idf_environment(work, idf_version):
+    idf_path = work / "esp-idf"
+    env = dict(
+        os.environ,
+        IDF_PATH=str(idf_path),
+        IDF_TOOLS_PATH=str(work / "idf-tools"),
+        IDF_PYTHON_ENV_PATH=str(work / "idf_python_env"),
+    )
+    # windows_job runs under Git Bash, whose MSYSTEM makes idf_tools.py refuse to run.
+    env.pop("MSYSTEM", None)
+    if not idf_path.exists():
+        run(
+            ["git", "-c", "core.longpaths=true", "clone", "--depth", "1"]
+            + ["--branch", idf_version, "--recurse-submodules", "--shallow-submodules"]
+            + ["https://github.com/espressif/esp-idf.git", idf_path],
+            env,
+        )
+    idf_tools = [sys.executable, idf_path / "tools/idf_tools.py"]
+    run(
+        idf_tools + ["install", "--targets", "esp32s3", "required", "cmake", "ninja"],
+        env,
+    )
+    # install-python-env fails if IDF_PYTHON_ENV_PATH doesn't exist yet.
+    Path(env["IDF_PYTHON_ENV_PATH"]).mkdir(exist_ok=True)
+    run(idf_tools + ["install-python-env"], env)
+    exported = subprocess.run(
+        [str(c) for c in idf_tools + ["export", "--format", "key-value"]],
+        env=env,
+        check=True,
+        capture_output=True,
+        text=True,
+    ).stdout
+    for line in exported.splitlines():
+        key, _, value = line.partition("=")
+        if key == "PATH":
+            value = value.replace("%PATH%", env["PATH"]).replace("$PATH", env["PATH"])
+        if key:
+            env[key] = value
+    scripts = "Scripts/python.exe" if os.name == "nt" else "bin/python"
+    return env, Path(env["IDF_PYTHON_ENV_PATH"]) / scripts
+
+
+def main(argv):
+    parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+    parser.add_argument(
+        "--inputs", type=Path, required=True, help="prepare's directory"
+    )
+    parser.add_argument("--work-dir", type=Path, required=True)
+    parser.add_argument("--idf-version", default="v6.1")
+    args = parser.parse_args(argv)
+    work = args.work_dir.resolve() / args.idf_version
+    work.mkdir(parents=True, exist_ok=True)
+    inputs = args.inputs.resolve()
+    env, idf_python = idf_environment(work, args.idf_version)
+    env["IDF_TARGET"] = "esp32s3"
+    version = (inputs / "version").read_text().strip()
+    component = work / "executorch"
+    shutil.rmtree(component, ignore_errors=True)
+    with tarfile.open(inputs / "dist" / f"executorch_{version}.tgz") as archive:
+        archive.extractall(component, filter="data")
+
+    models = ";".join(
+        (inputs / "models" / f"{m}.pte").as_posix() for m in ("smoke", "mul")
+    )
+    common = [
+        idf_python,
+        Path(env["IDF_PATH"]) / "tools/idf.py",
+        "-C",
+        ET_ROOT / "examples/espressif/idf_component_smoke",
+        f"-DET_COMPONENT_DIR={component.as_posix()}",
+        f"-DET_SMOKE_MODELS={models}",
+    ]
+    bundled = work / "bundled"
+    run(
+        common
+        + ["-B", bundled, f"-DSDKCONFIG={(bundled / 'sdkconfig').as_posix()}"]
+        + [f"-DET_SMOKE_KERNELS={(inputs / 'models/smoke_mul_kernels').as_posix()}"]
+        + ["reconfigure"],
+        env,
+    )
+    automatic = work / "automatic"
+    run(
+        common
+        + ["-B", automatic, f"-DSDKCONFIG={(automatic / 'sdkconfig').as_posix()}"]
+        + [f"-DEXECUTORCH_PYTHON={Path(sys.executable).as_posix()}", "reconfigure"],
+        env,
+    )
+
+    def selection(bundle):
+        kernels = json.loads((bundle / "kernels.json").read_text())
+        return {k: kernels[k] for k in ("models", "libraries", "prim_ops", "backends")}
+
+    made = selection(automatic / "executorch_kernels")
+    expected = selection(inputs / "models/smoke_mul_kernels")
+    if made != expected:
+        sys.exit(
+            f"FAIL: automatic mode selected {made}, but prepare's bundle has {expected}"
+        )
+    print("PASS: idf_component_smoke configured with a bundle and in automatic mode.")
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])

.ci/scripts/test_espressif_idf_consumer.sh

@@ -132,8 +132,7 @@ sources="${build_dir}/consumer sources"
 rm -rf "${sources}"
 mkdir -p "${sources}/idf_component_smoke"
 cp -r examples/espressif/idf_component_consumer "${sources}/"
-cp -r examples/espressif/idf_component_smoke/main \
-    examples/espressif/idf_component_smoke/sdkconfig.defaults "${sources}/idf_component_smoke/"
+cp -r examples/espressif/idf_component_smoke/main "${sources}/idf_component_smoke/"
 project="${sources}/idf_component_consumer"
 firmware_root="${build_dir}/consumer builds"