Building makemore Part 4: Becoming a Backprop Ninja

!!!!!!!!!!!!
I recommend you work through the exercise yourself but work with it in tandem and whenever you are stuck unpause the video and see me give away the answer. This video is not super intended to be simply watched. The exercise is here:
!!!!!!!!!!!!

Links:

Supplementary links:

Chapters:
00:00:00 intro: why you should care & fun history
00:07:26 starter code
00:13:01 exercise 1: backproping the atomic compute graph
01:05:17 brief digression: bessel’s correction in batchnorm
01:26:31 exercise 2: cross entropy loss backward pass
01:36:37 exercise 3: batch norm layer backward pass
01:50:02 exercise 4: putting it all together
01:54:24 outro

[Davourflave]

I can say without a doubt that there are not many highly qualified, passionate teachers who are also able to teach their subject. Sharing knowledge in this way is the greatest gift a researcher can give to the world! Me and everyone else thank you for that! :)

[BradCordovaAI]

Andrej you are a gifted teacher. I love this teaching style of starting from scratch with a simple specific model to set the structure and ideology of the problem. 2. Add necessary and motivated complexity to get where we are today, 3. Seamlessly transfer to modern technology (eg PyTorch) to solve modern problems. 4. You make it all simple and compress it into the essentials without unnecessary lingo. It reinvigorates my passion for the field. Thank you very much for taking so much time to make this for free for everyone.

[cojocarucosmin]

Bro just want to say that for the past 3 years I've been looking everywhere on the Internet for an explanation like thsi for backpropagation.. Found all kinf of things(e.g. Jacobian differentiable) but none actually made sense until today. U r the best, you bring so much value and let others light their candles at your light <3

[kshitijbanerjee]

These lectures are literally GOLD. I'd pay for these, but Andrej is kind enough to give everything for free.
I hope others find these gold lectures. Thank you so much for doing this. Please don't lose steam and I hope you continue to create them.

[weystrom]

Man, what a time to be alive. Imagine how hard it would be to get this kind of information just a couple decades ago. And now it's free and easily accessible at any convenient time.

Thank you, Andrey, truly.

[oferz]

Each time I finish a lecture in these series I feel satisfaction and I say to myself Andrej must be a wizard! he takes you by the hand from scratch showing you not only clean solution but shade light on problems and guide you through to the end. Remarkable talent.

[Raix]

I almost completed Exercise 1 all on my own, but I had to step back for a day to refresh the basics because my college algebra was a bit rusty from 10 years of not using it. Exercises 2 and 3 totally overwhelmed me. However, when I follow your explanations, I understand everything. This is a huge because I remember that professors at my college couldn't explain complex concepts so easily. Andrej, you are a gift to this world! <3

[nova]

I spent almost a whole day digesting this video. It's definitely worth it!

[DanteNoguez]

I was "taught" calculus in high school but didn't really understand anything at all. Now, after seven years of no math formal education at all, I was able to immediately understand this exercise thanks to your lecture on micrograd. You're a brilliant teacher and I'm really grateful for that!

[efogleman]

This lecture series is excellent. Seriously, some of the best learning resources for Neural Networks available anywhere: up-to-date, and goes deep into the details. These lectures with detailed examples and notebooks are an amazing resource. Thanks so much for this, Andrej.

[Themojii]

Hello Andrej, I truly love this approach that you included exercises in your video. Your suggestion to first attempt to solve the exercises and then watching as you provide the solutions is the most effective way I personally grasp the concepts. Thank you for your outstanding work!

[dohyun]

i am still on part 2 but i had to write this comment, your part 4 thumbnail is awesome and funny
I am very grateful for these lectures.
I could feel that the artificial intelligence knowledge that was intertwined inside me was well aligned because of you.

[andonisudupe]

yes, I always wanted to be a backprop ninja, now my dream will become true, thanks Andrej!

[Sickkkkiddddd]

Bruh, I'd be paying a shit ton of money in education for this otherwise free knowledge if it wasn't for your videos. Thank you so much, man. I cannot believe the ease with which you explain what seemed complex to me from a distance years ago. I cannot even believe I understand this stuff, man.

[kapitan]

Andrej, you are the best techer. I am 100% sure these lectures will become a CORE watching for any student who starts his ML journey. Hope we will have such lectures in CV and RL.

[joneskin]

I did my best to do exercise 1 on my own but couldn't really get anything on my own until `dh`. I got the through the rest with the video paused. The first day I spent a lot of time just repeating the patterns of calculations Andrej was doing, but I didn't really "get" what we were doing, and truly not for lack of trying, it just didn't click. The next morning it not only made perfect sense but seemed super obvious. I love how this course forces me to understand the theory, and how it pushes me to the edge of my limited ability.

[borismeinardus]

Andrej is providing the world with so much value, be it through his professional work in the industry (e.g. Tesla AI) or through education. He is literally one of the greatest of all time but is so down to earth and such a sweetheart.
Thank you very much for your hard work to make it easier for all the rest of us and for inspiring us! 💚

[RebeccaBrunner]

Thank you for providing a series that's so approachable but doesn't shy away from explaining the details. Also love the progression through all the impactful papers

[kishantripathi]

no words to explain my feelings. karpathy is just

[peterszilvasi]

I really appreciate the lectures that you share with us. It is not about definitions, raw memorization, or even exercise per se. Instead, first-principle-thinking: take a big "mess" and then broke down into small manageable pieces. You do not solely demonstrate the problem-solving approaches brilliantly but also ignite curiosity to dig deeper (to go down to the level of atoms) into a specific topic. Thank you for the preparation, the passion, and the memes! :D