Dive Into Deep Learning, Lecture 2: PyTorch Automatic Differentiation (torch.autograd and backward)

In this video, we discuss PyTorch’s automatic differentiation engine that powers neural networks and deep learning training (for stochastic gradient descent). In this section, you will get a conceptual understanding of how autograd works to find the gradient of multivariable functions. We start by discussing derivatives, partial derivatives, and the definition of gradients. We then discuss how to compute gradients using requires_grad=True and the backward() method. Thus, we cover classes and functions implementing automatic differentiation of arbitrary scalar-valued and non-scalar-valued functions. We also discuss the Jacobian matrix in PyTorch. Differentiation is a crucial step in nearly all machine learning and deep learning optimization algorithms. While the calculations for taking these derivatives are straightforward, working out the updates by hand can be a painful and tedious task.

#Autograd #PyTorch #DeepLearning

[adnanhashem]

In my opinion, what makes this a great video is its (quick) review of the prerequisite knowledge, and including simple examples to that we can compute by hand and then verify that we actually get the same thing when using PyTorch.

Thanks a lot!

[michaelscience]

This is what I was searching for months. I m so thankful Dr. Science for breaking down this concept for us

[sktdebnath]

This is what a proper tutorial should be. Thanks a lot. Subscribed

[reddysekhar]

Truly thankful to you. To the point without confusion. Thank you once again

[atanudasgupta]

Excellent videos and the textbook, deeply admire your contributions

[edd-bpmj]

why do you need to multiply v vector to the jacobian matrix? and what is v vector exactly?

[nitinsrivastav]

thanks man i was finding it extremely difficult to understand the maths behind backward and detach (although i have doen it in my high school) because no one was explaining them in this you 😍😍

[azzyfreeman]

Thanks for the video, it really cleared up Pytorch autograd, now I will be making notes on this gold nugget

[parisahajibabaee]

Amazing as always!! Very helpful and valuable video!

[홍성의-iy]

14:50 x.grad contains the valuesd of partial{y} / partial{x}
17:50 x.grad.zero_()
25:00 gradient for multiple inputs -> multiple outputs. Since the Jacobian is a matrix, we need to input a 1-d tensor to get a valid vector-output. => But our loss function has been a scalar, so this is why I am not accustomed to this form.
34:10 explaining .detach(). => treat those as constants, not a variable that we differentiate w.r.t.

[ГеоргийЧерноусов-оъ]

Nice explanation. Found it from youtube recomendations

[ItahangLimbu]

24:10
y.backward(torch.ones(3))
the input to y.backward is upstream gradient
and the size of the upstream gradient should be same as the size of y

and in previous cases we dont need insert upstream gradient cause
y, backward() same as y.backward(torch.tensor([1]))

[Gibson-xnxk]

First of all, I want to express my gratitude to you for the work you have done. There is one thing i want you to ask: why do we write the partial derivatives of the scalar function in the form of column? Whereas, following the logic of Jacobian matrix, it should be a row. Thanks in advance!

[asfandiyar]

Thank you sooo much! This finally clicked for me.

[rohithpeesapati]

thank you very much for uploading this video! very helpful!

[nicolaemaria]

great explanation! I hope it will get the likes it deserves

[arpit]

hi !! thanks for the wonderfull video :) can you please explain why vector v is 1?? and what is derivative w.r.t to self??

[ramincybran]

thnks myfriend - in last section - whats is the z.sum ? what is the the (SUM) function for ? why yyou put the sum ?

[callpie]

Please Do cover a playlist on Graph Neural Networks (at least discuss all the basics and methods of GNNs). The internet world lacks quality contents on this topic

[greender]

29:49 - As far as I understand in 'a.grad' should turn out [12., 18.]