[JakeStevens]

Summary:

etdump_cpu runs a CPU delegate model through a traced runner with ET_CPU_ETDUMP, drops warmup samples, and writes profile.json with median/min/max latency per plan step and XNNPACK operator, plus each XNNPACK operator's input and output shapes. Each CPU delegate's plan is reassembled from the runner log, so a program with several CPU delegates is profiled as a whole. Events with an unexpected sample count are reported with a warning, and malformed operator metadata degrades to no shapes. It fails if any plan step or, for XNNPACK regions, operator events are missing from the ETDump.

The model runner gains configurable warmups (ET_CPU_WARMUP_RUNS), timed iterations (ET_CPU_TIMED_RUNS), golden tolerances (ET_CPU_ATOL, ET_CPU_RTOL) and a latency summary. The workflows README now covers export and profiling.

Authored with Claude Code.

Differential Revision: D128598106

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24335

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (4 Unrelated Failures)

As of commit b2f15d3 with merge base 59fc134 ():

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Test Vulkan Backend (specialized runners) / test-model-vulkan-e2e / linux-job (gh) (trunk failure)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• trunk / test-arm-backend-vkml (test_pytest_models_vkml) / linux-job (gh) (trunk failure)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• trunk / test-cortex-m-ops / run (cortex-m0plus) / cortex-m-ops-cortex-m0plus (gh) (trunk failure)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 4
• trunk / test-cortex-m-ops / run (cortex-m7) / cortex-m-ops-cortex-m7 (gh) (trunk failure)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 4

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[meta-codesync]

@JakeStevens has exported this pull request. If you are a Meta employee, you can view the originating Diff in D128598106.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[shoumikhin]

@claude review effort=max

[claude]

Claude is reviewing PR #24335

• Gather PR context and diff against origin/main
• Independent reviews: runtime and plan tracing, XNNPACK profiler and build, workflow and runner
• Consolidate and deduplicate findings
• Fact-check every finding against the code
• Line-level notes: stale comments, naming, test cases
• Post the final review (re-checking the line numbers cited in fix links)

View job run

[JakeStevens]

@claude review effort=max

[claude]

Claude finished @JakeStevens's task in 48m 57s —— View job

---

PR Review: #24335

This PR adds an ETDump profiling workflow for CpuBackend. I recommend Request Changes: the new summarizer misattributes events across CPU delegates, can silently drop steps from its totals, and rebuilds the C++ step names from a scraped Info log, and there are smaller CI, Buck and link-compatibility problems.

Detailed review

Summary

The PR adds backends/cpu/workflows/etdump.py (etdump_cpu). It runs the CPU test runner with an ETDump and summarizes latency per step and per XNNPACK operator. The diff against 59fc134 also contains db71d80 (#23926), so this review covers both commits. The shallow checkout has no merge base with origin/main. db71d80 does three things:

• records one delegate event per CPUPlan step;
• adds XNNPACK operator events, with optional shapes;
• moves XNNProfiler into a new xnnpack_profiler library.

The summarizer gets profiles with several delegates wrong and can drop steps without failing. The CI, Buck and link changes need small fixes.

Not verified:

• describe_operators() (XNNPACKProvider.cpp:451-462) assumes that xnn_get_runtime_profiling_info reports exactly the opdata entries whose operator_objects[0] is set, in the same order. set_operator_metadata only checks the count. The XNNPACK submodule isn't checked out here, so I couldn't compare this filter with XNNPACK's.
• The internal mapping of XNNPACK_internal_headers to fbsource//xplat/third-party/XNNPACK:public-headers isn't visible from OSS. Builds with -c executorch.cpu_xnnpack_operator_shapes=true need that target to expose src/xnnpack/subgraph.h.

Code Quality

• The summarizer rebuilds step names in Python and reads the plan from a scraped Info log.
  • step_name (etdump.py:42-45) re-implements the trace_name construction in CPUPlan::select() (CPUPlan.cpp:392-399).
  • The plan comes from the CPU_PLAN_CHUNK Info lines (CPUBackend.cpp:259). Until now only the test helper at test_runtime.py:39 parsed them.
  • If the C++ naming changes, every step counts as missing and the run fails. Neither test would catch that, because CPUPlanTest.cpp:952 and test_etdump.py:22 each hard-code their own expected name.
  • A runner built with ET_LOG_ENABLED=0, or with a minimum log level above Info, fails with "Runner log has no complete CPU plan trace" even though the ETDump is complete.
  • Fix: emit trace_name from CPUPlan::describe() and read it instead of step_name, or tag step events so the ETDump alone identifies them. Fix this →
• summarize decides an event's kind from its sample count, and demotes any mismatch to init with only a warning (etdump.py:90-101).
  • The missing-steps check (:119) still counts a demoted step as present, so the step can drop out of totals_ms["step"] without the run failing.
  • main() doesn't validate --warmup or --iterations (:144-145), but the runner clamps ET_CPU_TIMED_RUNS to at least 1 (runner.cpp:47-48). With --iterations 0 the runner still does one timed run, so every event becomes init. The run then fails with the misleading "ETDump lacks XNNPACK operator events", or reports zero totals if the plan uses only Reference.
  • With --iterations 1, events with a single sample, such as Method::init, are reported as runtime.
  • Fix: reject a negative --warmup and an --iterations below 1, and check steps against rows whose kind is step. Fix this →
• Reference kernel events are labeled runtime and left out of the provider total.
  • backends/cpu/README.md:172-174 says events inside a step come from the provider, including Reference's native_call_<op> events. The Reference provider passes the tracer to its kernels (ReferenceProvider.cpp:1208).
  • These events have no delegate debug id, so the Inspector sets is_delegated_op to false (_inspector.py:524). summarize then labels them runtime, the same as Method::execute (etdump.py:102-103).
  • As a result, for plans with Reference steps the "provider operator total" counts only XNNPACK operators.
  • Fix: count these events as provider events, or say in workflows/README.md that the provider total covers only XNNPACK. Fix this →
• With no warmup, the first timed run also includes loading the method.
  • run_against_golden never loads the method, and Module::method_meta loads only the program (extension/module/module.cpp:663-670).
  • So the first forward (runner.cpp:59) also runs delegate init: plan selection, XNNPACK runtime creation and weight packing. With ET_CPU_WARMUP_RUNS=0, the runner's Latency: line includes that time.
  • Fix: call module.load_method("forward") before the loop. Fix this →

Infrastructure

• CI no longer builds the CPU backend in its default configuration.
  • .ci/scripts/test-native.sh:23-31 turns on devtools, the event tracer and operator shapes for the whole cpu component. native_cpu_engine_test and run_classification_cpu are now built only with the tracer.
  • The other CPU job, test-backend-cpu.yml, builds through the pybind preset, which also turns the tracer on for Linux and macOS (tools/cmake/preset/pybind.cmake:51,85).
  • No CI job builds CpuBackend with the tracer off, which is what a plain -DEXECUTORCH_BUILD_CPU=ON build gives users. So the tracer-off side of the new #ifdef ET_EVENT_TRACER_ENABLED blocks (CPUPlan.cpp:392, XNNPACKProvider.cpp:358) and cpu_xnnpack's calls into the XNNProfiler stubs are never compiled.
  • Fix: keep the default build, and build cpu_plan_test with the tracer in a separate configuration. Fix this →
• Buck now passes XNNPACK on to everything that depends on xnnpack_backend.
  • backends/xnnpack/targets.bzl:67 puts :xnnpack_profiler in exported_deps, and that target exports third_party_dep("XNNPACK") and event_tracer.
  • Before, XNNPACK was a private dep (:74). The only exported header, XNNPACKBackend.h, includes nothing from the profiler, so dependents now get XNNPACK's headers and XNNProfiler.h for no reason. CMake links the profiler PRIVATE.
  • Fix: move it to deps. Fix this →

Testing

• test_etdump.py never uses the real Inspector, and several summarize paths are untested.
  • Every event is a SimpleNamespace stand-in (backends/cpu/test/test_etdump.py:34-49).
  • Untested: the "ETDump lacks XNNPACK operator events" error, the runtime kind, input with several plans, and run().
  • Like the other backends/cpu/test Python tests, the file isn't in the pytest.ini testpaths, so OSS CI never runs it.
  • Fix: write a small ETDump with serialize_to_etdump_flatcc (devtools/etdump/serialize.py:91) and pass it through the real Inspector. Fix this →

API Design

• XNNPACK operators are numbered per delegate, but the README and summarize treat the numbers as per method.
  • backends/cpu/README.md:175-176 says <n> "counts operators of that name across all XNNPACK regions of the method".
  • The counter is XNNState::operator_counts (XNNPACKProvider.cpp:266), and each XNNProvider owns its own (:1393). Every delegate builds its own provider configuration (CPUBackend.cpp:90). So a method with two CPU delegates emits Convolution (NCHW, F32) #1 once for each delegate.
  • summarize merges every plan's regions into one list (etdump.py:81-83). profile.json then has rows with the same name and no indication of which delegate each came from.
  • The XNNPACK check (:122-125) passes as long as any one delegate produced operator events.
  • decode_plans and its test handle several plans, so this case is meant to work.
  • Fix: correct the README. Then either attribute each row to its delegate and run the step and XNNPACK checks per plan, or reject logs with more than one plan. The Inspector already separates events from different delegate calls by instruction id. Fix this →

Backward Compatibility

• Builds that link libxnnpack_backend.a by path now also need libxnnpack_profiler.a.
  • XNNProfiler.cpp moved out of xnnpack_backend into the new xnnpack_profiler archive (backends/xnnpack/CMakeLists.txt:136, linked PRIVATE at :168).
  • Out-of-tree builds that link the installed archives directly, as docs/source/using-executorch-faqs.md:76 described before this PR, fail with undefined XNNProfiler symbols until they add the new archive. Consumers using CMake targets or find_package(executorch) are unaffected.
  • Fix: mention this in the PR description and the release notes.

Specific Comments

• backends/cpu/test/runner.cpp:109: latencies[latencies.size() / 2] is the upper of the two middle samples. With the default 10 iterations, the runner's median_ms therefore differs from the statistics.median value in profile.json.
• backends/cpu/workflows/README.md:52-53: $CPU_ETDUMP and $CPU_TRACED_RUNNER are never defined; they are presumably the two outputs listed by --show-full-json-output. :runner also has no CMake target, so the section should say it is fbsource-only, as line 12 does.

Recommendation

Request Changes

The per-delegate numbering (API Design) and the sample-count classification (Code Quality) let the new workflow report misleading or incomplete profiles without failing. Both are small fixes. Emitting trace_name from the plan description would also remove the duplicated naming. The other findings are minor.