PyTorch Autograd Explained - In-depth Tutorial

In this PyTorch tutorial, I explain how the PyTorch autograd system works by going through some examples and visualize the graphs with diagrams. As you perform operations on PyTorch tensors that have requires_grad=True, you build up an autograd backward graph. Then when you call the backward() method on one of the output nodes, the backward graph gets traversed, starting at the node that the grad_fn attribute of the output node points to, and traversing backwards from there, accumulating gradients until leaf nodes are reached. The final leaf node gradients will be stored on the grad attribute of the leaf tensors.

This is my first PyTorch tutorial video. If you'd like to see more PyTorch related videos, let me know in the comments. And if you have anything specific about PyTorch that you would like me to make videos about, let me know.

Join our Discord community:

Connect with me:

[weebchina]

This is simply amazing man. It should be included as part of the official guide.

[wanhonglau]

this explanation is in-depth and original! can't find any other resources that explain pytorch autograd more in-depth than this

[zilianglin]

This video is really helpful. It teaches me not only the presupposition of the 60 minute Blitz but also an overview of the computational/backward graph.

[jiawei]

This is awesome. I've been trying to understand the autograd of PyTorch via bared codes for a whole week and I think I learned more in the 10 minutes of your video. Thank you and keep creating.

[chienyao]

Thank you for this wonderful explaination! Looking forward to your subsequent PyTorch videos.

[ishanmishra]

Been searching the net for a while about autograd but this is one of the most intuitive explanation with an awesome walkthrough. Thanks!

[egecnargurpnar]

Elliot, that was an marvelous video. You are explaining such a natural way and fast that you implemented that function. Thank you :)

[KSK]

Nice, crisp and clear explanation. Thanks for sharing this knowledge.

[jingbolin]

From Pytorch Forums, nice and detailed explanation for Pytorcher!!!

[rong]

Woudl be amazing if you make more pytorch this is simply the best explanation of pytorch autograd

[bitdribble]

I could finally understand a bit how this works. Thank you.

[GoodOldYoucefCef]

Thanks, your graphs made things clear to me!

[xzl]

Brilliant. I super like your metaphors of colors for "leaves", dried-up leaves and magics😍

[ravihammond]

This is the exact explanation I needed to clear up autograd. Many thanks from Australia!

[m.y.s]

Nicest explanation for autograd on the internet! Thx!

[gabrielcbenedito]

I'm so glad you decided to make this video... this explanation is just perfect! I just subscribed and hope to see more of those!

[muhammadroshan]

So much information in such short time. I am amazed just simply startled :o

[weiwang]

That's so great and clear! Thanks Elliot!

[AlexeyMatushevsky]

That's magical! Some much effort to explain simple concept! As a result - I really feel like I really understood it. Thanks! Schema's in the hands of the master - priceless!

[mohammadhassanvali]

A really fantastic and thorough explanation of autograd. Thanks a lot!