Separate forward and backwad compilation and support higher order derivatives for aot_function #856
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ghstack-poisoned]
anjali411
commented
Jun 7, 2022
…tion" [ghstack-poisoned]
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
commented
Jun 7, 2022
functorch/_src/aot_autograd.py
Outdated
| @@ -140,12 +140,15 @@ def create_aot_autograd_function( | |||
| compiled_fw = None | |||
| compiled_bw = None | |||
| num_outs = None | |||
| joint_inputs = None | |||
There was a problem hiding this comment.
need to save these tensors in the context
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
zou3519
reviewed
Jun 9, 2022
functorch/_src/aot_autograd.py
Outdated
Comment on lines
217
to
220
| func_code = bw_module.code.split('self, ') | ||
| # print(func_code[0] + func_code[1]) | ||
| exec(func_code[0] + func_code[1], globals()) | ||
| f = create_aot_autograd_function(forward, bw_compiler, bw_compiler, partition_fn, aot_decompositions, grad_state) |
There was a problem hiding this comment.
Two questions:
- Why are we passing
forwardto create_aot_autograd_function? I would have expected us to pass bw_module.code without the self argument - What is the exec for? Are you trying to test this without the
create_aot_autograd_functionline?
There was a problem hiding this comment.
- forward is the name of the function generated by running
bw_module.code - exec executes the bw_module.code to create a backward function which is the forward for the next pass
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
changed the title
anjali411
changed the title
…r order derivatives for aot_function" Test Plan: Existing tests should pass [ghstack-poisoned]
…r order derivatives for aot_function" Test Plan: Existing tests should pass [ghstack-poisoned]
…r order derivatives for aot_function" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
commented
Jul 15, 2022
| def fake_fn(primals, tangents): | ||
| return fx_g_b(primals, tangents) | ||
| fx_g_b = make_fx(functionalize(fake_fn))(flat_tensor_args, inp_grad_outs) | ||
| saved_value_nodes = _get_saved_values(fx_g_b, saved_value_names) |
There was a problem hiding this comment.
Unfortunately this approach doesn't always work because the newly generated fx graph may not have the same nodes as the previous graph. We need an alternate way to select nodes of interest in this new graph!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
Test Plan: Existing tests should pass