Skip to content

Add in-process entry points to gen_all_oplist and codegen.gen - #24704

Merged
rascani merged 1 commit into
esp-idf-generator-hygienefrom
codegen-in-process-entrypoints
Nov 9, 2026
Merged

rascani merged 1 commit into
esp-idf-generator-hygienefrom
codegen-in-process-entrypoints

Conversation

@rascani

@rascani rascani commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

gen_oplist, gen_selected_prim_ops and gen_max_kernel_num each have a
function their main() calls, but gen_all_oplist and codegen.gen do all
their work in main(), so a Python caller has to start a subprocess or
fake a command line to use them. Their work moves into
gen_all_oplist.gen_all_oplist() and codegen.gen.generate(), and main()
keeps only the command-line handling: argument parsing, the @-prefixed
list file, the BUCK checks and gen_all_oplist's single-yaml mode, which
still leaves debug_info alone. generate() takes main()'s flags as keyword
arguments (outputs is --generate; the unused --static-dispatch-backend is
left out) and builds its FileManager directly instead of through
make_file_manager(options). Its tags.yaml and native_functions.yaml
default to the copies packaged with torchgen, the ones
tools/cmake/Codegen.cmake passes, so callers need not know where torchgen
keeps them; --tags-path and --aten-yaml-path default the same way.

The command lines behave as before: a selective build of two models plus
an op list makes byte-identical sources with the old and new tools. The
ESP-IDF component's gen_kernels.py calls both functions in the next
change.

Authored with Claude Code.

@pytorch-bot

pytorch-bot Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24704

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 7758683 with merge base 4411b8d (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@rascani
rascani added this pull request to stack #24713 November 5, 2026 23:48
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@rascani
rascani force-pushed the codegen-in-process-entrypoints branch from acdb002 to 6a1e995 Compare November 6, 2026 19:19
@rascani
rascani force-pushed the codegen-in-process-entrypoints branch from 6a1e995 to 54a5479 Compare November 6, 2026 20:40
@rascani
rascani marked this pull request as ready for review November 6, 2026 20:40
@rascani
rascani requested a review from JakeStevens November 6, 2026 20:41
@rascani
rascani force-pushed the codegen-in-process-entrypoints branch from 54a5479 to 164f8f6 Compare November 6, 2026 21:15

@JakeStevens JakeStevens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fyi claude is nervous about internal ci with this, keep an eye i guess

@rascani
rascani force-pushed the codegen-in-process-entrypoints branch from 164f8f6 to b0e7c0b Compare November 9, 2026 17:32
gen_oplist, gen_selected_prim_ops and gen_max_kernel_num each have a
function their main() calls, but gen_all_oplist and codegen.gen do all
their work in main(), so a Python caller has to start a subprocess or
fake a command line to use them. Their work moves into
gen_all_oplist.gen_all_oplist() and codegen.gen.generate(), and main()
keeps only the command-line handling: argument parsing, the @-prefixed
list file, the BUCK checks and gen_all_oplist's single-yaml mode, which
still leaves debug_info alone. generate() takes main()'s flags as keyword
arguments (outputs is --generate; the unused --static-dispatch-backend is
left out) and builds its FileManager directly instead of through
make_file_manager(options). Its tags.yaml and native_functions.yaml
default to the copies packaged with torchgen, the ones
tools/cmake/Codegen.cmake passes, so callers need not know where torchgen
keeps them; --tags-path and --aten-yaml-path default the same way.

The command lines behave as before: a selective build of two models plus
an op list makes byte-identical sources with the old and new tools. The
ESP-IDF component's gen_kernels.py calls both functions in the next
change.

Authored with Claude Code.
@rascani
rascani force-pushed the codegen-in-process-entrypoints branch from b0e7c0b to 7758683 Compare November 9, 2026 18:37
@rascani
rascani merged commit 1e03c8d into main Nov 9, 2026
210 checks passed
@rascani
rascani deleted the codegen-in-process-entrypoints branch November 9, 2026 21:06

This branch was previously deployed

1 inactive deployment
cadence — 7758683b Deployed Nov 9, 2026 by rascani via hifi-op-test / hifi4 #37400
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants