Backpropagation calculus | Chapter 4, Deep learning

An equally valuable form of support is to share the videos.

This one is a bit more symbol-heavy, and that's actually the point. The goal here is to represent in somewhat more formal terms the intuition for how backpropagation works in part 3 of the series, hopefully providing some connection between that video and other texts/code that you come across later.

For more on backpropagation:

Music by Vincent Rubinetti:

Thanks to these viewers for their contributions to translations
Hebrew: Omer Tuchfeld

------------------
Video timeline
0:00 - Introduction
0:38 - The Chain Rule in networks
3:56 - Computing relevant derivatives
4:45 - What do the derivatives mean?
5:39 - Sensitivity to weights/biases
6:42 - Layers with additional neurons
9:13 - Recap
------------------

Various social media stuffs:

[bluebrown]

Two things worth adding here:
1) In other resources and in implementations, you'd typically see these formulas in some more compact vectorized form, which carries with it the extra mental burden to parse the Hadamard product and to think through why the transpose of the weight matrix is used, but the underlying substance is all the same.

2) Backpropagation is really one instance of a more general technique called "reverse mode differentiation" to compute derivatives of functions represented in some kind of directed graph form.

[thomasclark]

This series was my first introduction to Machine Learning 3 years ago. I now work full-time as an AIML Scientist, my life is forever changed. Thank you.

[kslm]

“The definition of genius is taking the complex and making it simple.”
- Albert Einstein

You are genius.

[cineblazer]

Dear Grant,

A year ago, I decided I wanted to learn Machine Learning and how to use it to make cool stuff. I was struggling with some of the concepts, so I went to YouTube and re-discovered this series on your channel.

Out of all the courses I've tried and all the hours of other content I've sat through, your videos stand out like a ray of sunshine. I just got my first full-time job as a Machine Learning Engineer, and I can confidently say it would never have happened without this series.

Your channel may have affected the course of my life more than almost any other. Thanks for all your hard work!

[noahkupinsky]

Hey for all of you getting discouraged because you don’t understand this - that was me last year. I went and taught myself derivatives and came back to try again and suddenly I understand everything. It’s such an amazing feeling to see that kind of work pay off. Don’t give up kiddos

[hiqwertyhi]

It's not that no-one else makes top-notch math/cs videos, it's that this guy makes it CLICK.

[PhilippeCarphin]

This series is insanely good. As a teacher, I feel like Salieri watching Mozart play and being like "It's so beautiful, how is he so good!"

[yashjindal]

I just started out with my ML career. This entire series made me feel as if I knew it all along. Thank you Grant
I will return to this comment to share my professional progress😊

[Mrrajender]

Many guys claim to know. Some guys actually know. But only one guy actually knows and can explain to his grandma as well with very beautiful animations. You are that ONE !!!

[suharsh]

This is the longest 10 minute video I have ever watched. Literally took me half an hour, but the feeling of the idea behind this completely settling in, makes it totally worth it!

[hutc]

Your work of making high levels of math accessible to anyone wishing to learn a variety of new topics is not obvious to me. You succeed to explain everything so clearly, making me want to start learning maths again, reminding me of and introducing me to beautiful aspects of math, and you deserve more than a 'thank you' :)

[SaifUlIslam-dbnu]

It has taken me about 3-4 days worth time to understand all of these 4 lectures, lectures which are in total, no longer than 1 hour and 30 minutes.



And I feel proud.

[shofada]

This is how 21st teaching should look like. It feels like your work should be made a "human right". Thank you.

[SaintKhaled]

The quality of this education is top-tier. I absolutely am speechless that you make it freely accessible. Thank you so much!

[cineblazer]

I'm taking Machine Learning by Andrew Ng on Coursera right now, and just got stuck on backpropagation. Thank you thank you thank you thank you Grant, you have no idea how incredibly helpful your videos are and how much your channel has inspired me through the years.

[snf]

At time when I just finished my university — I could not imagine that at one chilly Sunday evening, in almost 15 years after the graduation, I will sit with a bottle of beer, watch math videos, and have so much fun! Thank you!

[antoniobernardo]

this is easily the best channel in youtube today! once I get a job i will more than glad to support you!

[thiyagutenysen]

I came here after Andrew Ng's week 5 in coursera and you blew my mind

[vladimirfokow]

Thank you a lot for this series! It has really helped me get into this topic, and changed my life. Your intuitions have been immensely helpful in my efforts to understand backpropagation. I just can't overestimate, how great your channel is!

[alexdebate]

Grant, I've come back to this series many times over the last five years. Every time I do, I pick up more and more pieces of the puzzle. I think I've finally got it now, but we shall see! Thank you!