The Most Important Algorithm in Machine Learning

Shortform link:

In this video we will talk about backpropagation – an algorithm powering the entire field of machine learning and try to derive it from first principles.

OUTLINE:
00:00 Introduction
01:28 Historical background
02:50 Curve Fitting problem
06:26 Random vs guided adjustments
09:43 Derivatives
14:34 Gradient Descent
16:23 Higher dimensions
21:36 Chain Rule Intuition
27:01 Computational Graph and Autodiff
36:24 Summary
38:16 Shortform
39:20 Outro

USEFUL RESOURCES:

Jürgen Schmidhuber's blog on the history of backprop:

CREDITS:

[Mutual_Information]

Back prop is a hard, heavy thing to explain, and this video does it extremely well. I mean, that section 'Computational Graph and Autodiff' might be the best explanation of that subject on the internet. I'm very impressed - well done!

[CuriousLad]

Funnily enough, the calculus portion of the video is probably one of the best explained I've seen

[aminebouramada]

this's by far the most clearer explaination and simplification of backpropagation i have watched

[shikhargairola]

It’s probably the best explanation of backward propagation. Hats off to your hard work and saving this so valuable content.

[vastabyss]

It makes sense that you would cover both computational neuroscience AND machine learning since they both play a significant role in AI research. The sort of content you're making is definitely 3Blue1Brown level. Keep up the good work!

[undertheshadow]

"Wait, It's all derivatives?"
"Always has been"

Great work pal. Provides excellent clarity.
Looking forward to the second part.

[matheusmendonca]

By far the best ML explanation I have seen on internet.

[ReighKnight]

The visuals on this video is from another planet . So Good

[keithwallace]

This has to be one of the greatest explanation of the inner working of learning in ML, I love it!

[priteshtadvi]

I knew that calculus is important for machine learning but never knew that 12th grade derivatives are that much important.
When you said about chain rule, that bring me back to my school days, I never thought that derivatives, integration and probabilities will be used this way in future.
Well explained video.
Thanks for sharing this knowledge and conveying process much simply.

[black_crest]

This just might be the most underrated video on Back Propagation that I've ever seen! I hope more people come across this

[maheshwaransivagnanam]

I've been trying to get into ML for quite a while now. This is by far the best explanation of gradient descent and back propagation hands down!!!

Amazing work!!!

[TruthOfZ]

i just made that in python for a simple quadratic YOU !!!! i just learned python and machine learning
Using desired y=0 i could also find one solution of the equation... wow i love this so much!!
The only different i did was to make x the weight and not the coeficients which i wanted them to be fixed inputs

What you helped me realise is that any system that can put in a computational graph like that 30:04 ...it can be embeded backpropagation regardles
THANK YOU im out of words

Also when the next loss is bigger or equal than the preview loss after one iteration... i divided the learning rate by a factor of 2 or 10 for more accuracy and if the next loss was smaller than the preview one i multiple the learning rate by a factor of 1.1 to 1.5 to speed up the proccess...thus having results in hundreds or even thousands less generations/iterations and less time

I can use this for optimizing my desired outputs in any system !!! JUST WOW!!

[Alwaysiamcaesar]

I actually pictured this all in my head successfully where I thought I had everything in a canonical deep neural network figured out the other day. It’s one thing to hold it, it’s another to do the detailed, gritty work of explaining it in video format. Very well done.

[RedantRedant]

That was an outstanding explanation. Your ability to explain higher mathematical concepts in such simple terms is really an amazing service to the rest of us who wanna understand these subjects but don’t have a mathematics degree. Thank you.

[pradhumnkanase]

There could not have been a better explanation. Hats off to you

[bungerwow]

I've seen probably 20 videos on this and your explanation of the derivatives for someone not in calculus was really helpful. thanks.

[ram-myfl]

Most Comprehensive Explanation EVER

my opinion : better than
3b 1b, No offence to 3b 1b Hes great at it and one of the pioneers who did these kind kf visual explanations.

But i like your explanation as it is slow paced & comprehensive

[KMegahertz]

This is a visual masterpiece! Well done!

Much of this was a review for me as I took the time to go through all this last year. I did an implementation of the MNIST handwritten number neural network and had to learn all the calculus covered here to work out the backpropagation math. You really do have to dig in to it to get a good handle on it but it's fun stuff.

[Anonymous-frop]

Damn, I was wondering where you've been since over half a year, whilst I was stuck in backpropagation😂 and here you came back like a true mind reader. Glad to see you back❤