Visualizing gradients tutorial (issue #3186)

[j-silv]

Fixes #3186

Description

Add draft for visualizing gradients tutorial. Link is here but the content is old and the files need to be re-built.

Checklist

• The issue that is being fixed is referred in the description (see above "Fixes #ISSUE_NUMBER")
• Only one issue is addressed in this pull request
• Labels from the issue that this PR is fixing are added to this pull request
• No unnecessary issues are included into this pull request.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3389

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 4ab29bf with merge base ec4e137 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[sekyondaMeta]

Generally seems to be headed in the right direction in terms of tone and organization from my perspective.
Can you add perquisite knowledge for this.

[soulitzer]

Thanks for the working on this tutorial. Overall I'd say though that this section (prior to the actual visualizing gradients part) can be much shorter.

By the end of this tutorial, you will be able to:

Differentiate between leaf and non-leaf tensors

have a diagram from https://github.com/szagoruyko/pytorchviz, point to the leafs

Know when to use\ retain_grad vs. ``require_grad`

"use requires_grad for leaf, use retain_grad for non-leaf"

[j-silv]

Thank you for the comments, they were really helpful. Let me know if you think the first section is still too long.

Concerning the "visualizing gradients" section with an actual example, I'm not sure if I'm going about retaining the gradients for intermediate tensors correctly. My thought process was to use a forward hook, call retain_grad() on the output tensor of that module, and then store that output tensor in a list. Later, after calling loss.backward(), I could then pluck out the grad attribute of that tensor and plot it.

Initially I tried using a backward pass hook like register_full_backward_hook() but this didn't work because the ResNet model performs some inplace operations (i.e. ReLU and one += addition) and PyTorch complains about it:

RuntimeError: Output 0 of BackwardHookFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.

I know that I can plot the gradients for the parameters by just looping through the named_parameters() but I would like to also plot the gradients for the intermediate tensors.

If anyone sees a problem with my method let me know. The current state of the code isn't doing what I expected so I still have to debug it.

EDIT 1: I stumbled upon this issue. Perhaps it's better to switch to using tensor hooks as suggested by alban, instead of storing the outputs through a forward pass and then later accessing their .grad

EDIT 2: I decided to not use ResNet but instead a simplified fully connected network as explained in the BatchNorm paper. It is purely for educative purposes, but it actually shows the results I was expecting. With the ResNet implementation, I believe that the residual connections and ReLU non-linearity are muddying the negative effect on the gradients if they don't have BatchNorm. I'll push an updated PR sometime today.

[sekyondaMeta]

@soulitzer what are your thoughts on these new updates?

[soulitzer]

Thanks for the updates! Looking pretty good, but added some comments on wording, etc.

[j-silv]

@soulitzer here's my latest draft. Let me know if you would like anything changed!

[j-silv]

@soulitzer sorry to bug you as I know you are busy! When you get a chance can you review these changes? I'd love to get the tutorial merged :)

[soulitzer]

Sorry for the delay, and thanks for your patience addressing the comments!
Overall, I think this is quite good!
My only comment would be that it should actually be two separate tutorials, since now the second part is kind of unrelated to the first now.
Would you mind splitting it? Sorry for the churn here.

[j-silv]

Sorry for the delay, and thanks for your patience addressing the comments! Overall, I think this is quite good! My only comment would be that it should actually be two separate tutorials, since now the second part is kind of unrelated to the first now. Would you mind splitting it? Sorry for the churn here.

No worries I can do that!

What difficulty do you think there should be for each tutorial? Also should I open up a separate PR for one of the splits?

[soulitzer]

What difficulty do you think there should be for each tutorial? Also should I open up a separate PR for one of the splits?

Maybe beginner for both? WDYT

Also should I open up a separate PR for one of the splits?

Yes, thanks!

[j-silv]

Maybe beginner for both? WDYT

I took a look through most of the tutorials on the PyTorch docs and I think that beginner is reasonable. I could maybe see intermediate for the gradients one, but I'm on the fence. @sekyondaMeta do you think beginner for both?

[soulitzer]

could maybe see intermediate for the gradients one, but I'm on the fence.

Either is fine to me!