Gradient Descent From Scratch In Python

We'll learn about gradient descent, a technique for training neural networks. We'll then implement gradient descent from scratch in Python, so you can understand how it works. We'll implement gradient descent by training a linear regression model to predict the weather. In future videos, we'll build on this to create complex neural networks!

Chapters
0:00 Introduction
01:49 - Linear Regression Intuition
07:53 - Measuring Loss
15:28 - Parameter Updates
16:11 - Gradients And Partial Derivatives
23:29 - Learning Rate
28:35 - Implement Linear Regression
36:09 - Training Loop

This video is part of our new course, Zero to GPT - a guide to building your own GPT model from scratch. By taking this course, you'll learn deep learning skills from the ground up. Even if you're a complete beginner, you can start with the prerequisites we offer at Dataquest to get you started.

If you're dreaming of building deep learning models, this course is for you.

Best of all, you can access the course for free while it's still in beta!

Sign up today!

[rezNezami]

Hi Vik. Just want to say, from a Math / Physics/ and AI teacher, that your style of teaching simply flawless. You are a natural. Thank you.

[hchattaway]

Please keep doing these, they are really excellent!

[FootballIsLife]

Exactly at 19:44, you mention that the derivative of loss function regarding b is the same as loss function but I don't think so, because derivative of :

dL/db ( (wx+b) - y )^2 = 2((wx+b)-y)

and

dL/dw = 2x((ws+b)-y)

can anyone help me out ?

[fd]

Is there a discord to discuss the projects on this channel?

[mytechnotalent]

Great tutorial! Question, in your backward method, you did the optimization step did you not want to create an optimizer method for that as that is not technically part of the back propagation?

[zoeadah]

Can I please know where you got your clean whether dataset from?

[josuecurtonavarro]

Hi Vik! Thanks o much for the amazing work! Your content is always one of my best choices when it comes to learning DataScience and ML. I have a doubt though about the video in minute 40:56. You mention that in the init_params function, if we substract 0.5 from the result of np.random.rand(), it would rescale weights from -0.5 to 0.5. But wouldn't it just gives us (randomly) some negative values (depending also on the chosen seed) whenever the ones returned by np.random.rand() function are less than 0.5? Thanks so much again and please, keep on doing what you do! I've already come a long way thanks to all your work!

[yujin]

i think the derivative of the loss function should be 160×(80w+11.99−y), instead of 2×(80w+11.99−y)

[anfedoro]

finally I have managed to implement the gr descent for linear regression myself :-).. almost with no looking back to Vik's notebook. Can consider now that I understand how it works and all math underlying. Just curious, why my final weights and bias are very different compare to that sklean is calculating ? I plot all three - original test labels, calulated via my own procedures and calculated via sklearn.. I see that my is less acurate vs sklearn. Why it could be ?

[rjchen]

Thanks for the tutorial! Could you also add access to the data 'clean_weather.csv'

[AvinashChandrashukla]

What i am neet step by step bussiness analyst

[kurtisunknown]

This is the easy form of the gradient, how about when we have a difficult form of cost function ?

[kevinmccann]

I don't think you did the two derivatives correctly. Please explain.