Add a nightly matrix for the ESP-IDF component

nightly-espressif-idf.yml runs the component across every target,
ESP-IDF release and configuration that the per-PR jobs only sample. A
prepare job generates and packs the component and exports the smoke,
esp_nn:: mul, ESP-NN delegate and Cadence wake word models with their
kernel bundles, plus an all_ops bundle that registers every op of every
kernel library. Twelve build jobs (ESP-IDF 6.0.1, 6.1 and master, four
shards each) take that artifact and nothing else: they serve the archive
from a file:// mirror, list the idf_component_consumer configurations
with idf-build-apps (each sdkconfig.ci.* file for ESP32, -S3, -C3, -C5,
-C6 and -P4, filtered by .build-test-rules.yml), build their shard from
the mirror with ESP-IDF's own Python environment, check that every
setting of its sdkconfig.ci.* file reached the build's sdkconfig, since
ESP-IDF only warns about an unknown symbol, and run each build on QEMU
or esp-emu. Each job records the ESP-IDF commit it tested and uploads
its build and emulator logs to S3, where they stay 14 days instead of
GitHub's 2. The configurations are default (the smoke model alone);
nolog (logging compiled out), perf (ExecuTorch built without -Os, and
the project for speed) and debug (debug assertions, and full program
verification, which the app then runs on each model as it loads it),
which each run all four models, so the delegate and the quantized,
Cadence and esp_nn:: kernels run under each setting; wakeword, delegate
and oplib, each a model with its own bundle, on ESP32-S3 and -C6, since
the four-model configurations already run those models everywhere;
all_ops, also on ESP32-S3 and -C6, with an 8 MB flash layout, so every
vendored kernel compiles under ESP-IDF's flags and every registration
resolves when the image links; plus a build-only ESP32-P4 configuration
for silicon before rev3, which esp-emu does not run.

The consumer takes the model directory as ET_CONSUMER_MODELS_DIR, from
which CONFIG_ET_CONSUMER_MODEL picks a list of models and their combined
bundle. It has its own sdkconfig.defaults instead of overriding
SDKCONFIG_DEFAULTS. The script builds each listed configuration itself
with its own sdkconfig, because idf-build-apps keeps one sdkconfig in
the project directory, which the next build overwrites before the
emulator run needs it, and runs idf-build-apps from a venv of its own so
ESP-IDF's environment stays as ESP-IDF installs it.

Jobs on macOS and Windows configure the two-model idf_component_smoke
project with the packed component, once with its bundle and once in
automatic mode, and check that both select the same kernels. The
registry validation (compote component upload --dry-run
--allow-existing) runs here on the packed archive. setup.sh accepts an
ESP-IDF branch such as master. The workflow also runs for the
ciflow/nightly label, and on pull requests that change the matrix itself
(the workflow, its two scripts and idf_component_consumer/), since
workflow_dispatch only works from the default branch and
test-espressif.yml already builds and runs the packed component on every
pull request that can change it. Pull requests skip ESP-IDF master,
which breaks on its own schedule.

Authored with Claude Code.

Author: rascani

.ci/scripts/test_espressif_idf_configure.py

@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Configure idf_component_smoke with two models on this host.
+
+For the nightly macOS and Windows jobs. Installs ESP-IDF and its ESP32-S3 tools
+under --work-dir/<--idf-version> with ESP-IDF's own Python environment, then runs
+`idf.py reconfigure` on idf_component_smoke with the smoke and mul models and
+the component test_espressif_idf_nightly.sh prepare packed: once with the
+kernel bundle prepare made, and once in automatic mode, where this Python
+(which must import executorch) makes the bundle. Both bundles must select the
+same kernels.
+"""
+
+import argparse
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tarfile
+from pathlib import Path
+
+ET_ROOT = Path(__file__).resolve().parents[2]
+
+
+def run(cmd, env):
+    print("+", " ".join(str(c) for c in cmd), flush=True)
+    subprocess.run([str(c) for c in cmd], env=env, check=True)
+
+
+def idf_environment(work, idf_version):
+    idf_path = work / "esp-idf"
+    env = dict(
+        os.environ,
+        IDF_PATH=str(idf_path),
+        IDF_TOOLS_PATH=str(work / "idf-tools"),
+        IDF_PYTHON_ENV_PATH=str(work / "idf_python_env"),
+    )
+    # windows_job runs under Git Bash, whose MSYSTEM makes idf_tools.py refuse to run.
+    env.pop("MSYSTEM", None)
+    if not idf_path.exists():
+        run(
+            ["git", "-c", "core.longpaths=true", "clone", "--depth", "1"]
+            + ["--branch", idf_version, "--recurse-submodules", "--shallow-submodules"]
+            + ["https://github.com/espressif/esp-idf.git", idf_path],
+            env,
+        )
+    idf_tools = [sys.executable, idf_path / "tools/idf_tools.py"]
+    run(
+        idf_tools + ["install", "--targets", "esp32s3", "required", "cmake", "ninja"],
+        env,
+    )
+    # install-python-env fails if IDF_PYTHON_ENV_PATH doesn't exist yet.
+    Path(env["IDF_PYTHON_ENV_PATH"]).mkdir(exist_ok=True)
+    run(idf_tools + ["install-python-env"], env)
+    exported = subprocess.run(
+        [str(c) for c in idf_tools + ["export", "--format", "key-value"]],
+        env=env,
+        check=True,
+        capture_output=True,
+        text=True,
+    ).stdout
+    for line in exported.splitlines():
+        key, _, value = line.partition("=")
+        if key == "PATH":
+            value = value.replace("%PATH%", env["PATH"]).replace("$PATH", env["PATH"])
+        if key:
+            env[key] = value
+    scripts = "Scripts/python.exe" if os.name == "nt" else "bin/python"
+    return env, Path(env["IDF_PYTHON_ENV_PATH"]) / scripts
+
+
+def main(argv):
+    parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+    parser.add_argument(
+        "--inputs", type=Path, required=True, help="prepare's directory"
+    )
+    parser.add_argument("--work-dir", type=Path, required=True)
+    parser.add_argument("--idf-version", default="v6.1")
+    args = parser.parse_args(argv)
+    work = args.work_dir.resolve() / args.idf_version
+    work.mkdir(parents=True, exist_ok=True)
+    inputs = args.inputs.resolve()
+    env, idf_python = idf_environment(work, args.idf_version)
+    env["IDF_TARGET"] = "esp32s3"
+    version = (inputs / "version").read_text().strip()
+    component = work / "executorch"
+    shutil.rmtree(component, ignore_errors=True)
+    with tarfile.open(inputs / "dist" / f"executorch_{version}.tgz") as archive:
+        archive.extractall(component, filter="data")
+
+    models = ";".join(
+        (inputs / "models" / f"{m}.pte").as_posix() for m in ("smoke", "mul")
+    )
+    common = [
+        idf_python,
+        Path(env["IDF_PATH"]) / "tools/idf.py",
+        "-C",
+        ET_ROOT / "examples/espressif/idf_component_smoke",
+        f"-DET_COMPONENT_DIR={component.as_posix()}",
+        f"-DET_SMOKE_MODELS={models}",
+    ]
+    bundled = work / "bundled"
+    run(
+        common
+        + ["-B", bundled, f"-DSDKCONFIG={(bundled / 'sdkconfig').as_posix()}"]
+        + [f"-DET_SMOKE_KERNELS={(inputs / 'models/smoke_mul_kernels').as_posix()}"]
+        + ["reconfigure"],
+        env,
+    )
+    automatic = work / "automatic"
+    run(
+        common
+        + ["-B", automatic, f"-DSDKCONFIG={(automatic / 'sdkconfig').as_posix()}"]
+        + [f"-DEXECUTORCH_PYTHON={Path(sys.executable).as_posix()}", "reconfigure"],
+        env,
+    )
+
+    def selection(bundle):
+        kernels = json.loads((bundle / "kernels.json").read_text())
+        return {k: kernels[k] for k in ("models", "libraries", "prim_ops")}
+
+    made = selection(automatic / "executorch_kernels")
+    expected = selection(inputs / "models/smoke_mul_kernels")
+    if made != expected:
+        sys.exit(
+            f"FAIL: automatic mode selected {made}, but prepare's bundle has {expected}"
+        )
+    print("PASS: idf_component_smoke configured with a bundle and in automatic mode.")
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])

.ci/scripts/test_espressif_idf_consumer.sh

@@ -132,8 +132,7 @@ sources="${build_dir}/consumer sources"
 rm -rf "${sources}"
 mkdir -p "${sources}/idf_component_smoke"
 cp -r examples/espressif/idf_component_consumer "${sources}/"
-cp -r examples/espressif/idf_component_smoke/main \
-    examples/espressif/idf_component_smoke/sdkconfig.defaults "${sources}/idf_component_smoke/"
+cp -r examples/espressif/idf_component_smoke/main "${sources}/idf_component_smoke/"
 project="${sources}/idf_component_consumer"
 firmware_root="${build_dir}/consumer builds"