Neural Networks Demystified [Part 5: Numerical Gradient Checking]

When building complex systems like neural networks, checking portions of your work can save hours of headache. Here we'll check our gradient computations.

Supporting code:

In this series, we will build and train a complete Artificial Neural Network in python. New videos every other friday.

Part 1: Data + Architecture
Part 2: Forward Propagation
Part 3: Gradient Descent
Part 4: Backpropagation
Part 5: Numerical Gradient Checking
Part 6: Training
Part 7: Overfitting, Testing, and Regularization

@stephencwelch

[isaaclumpkins]

Just an amazing resource. There is NO reason we can't communicate complex concepts simply and not have to cut corners either. Thank you for this!

[bobsmithy]

I swear, one day I shall master this art.

[kariuki]

NO!!! I REFUSE TO BE INTIMITADED BY THIS!! I WILL GET IT!!... eventually

[SecondQuantisation]

I'm a mathematician and people who are either terrible teachers or who were taught by terrible teachers are the bane of my existence. Being able to clearly and methodically break down a problem or method into steps small enough to be clear and logically connected but not so numerous as to require 100 years to communicate is a very rare skill/gift. I've just watched this Neural Networks series of yours and I've got to say it's very good in its quality of presentation and, more importantly, quality of material. +1 subscriber!

just too good....now I can confidently say that yes I know Neural Networks and it's not a mysterious math anymore...Thanks a lot for these videos

[SethuIyer]

Wow, Now neural network is no more a mysterious black box to me. thank u

[hiprabhat]

oh man. I was watching Andrew Ng course on coursera and could not get everything. Your explanation is way better...

[chrisa]

These videos are awesome! I'm taking master classes on machine learning and these videos help bring things into perspective whenever I get lost during parts of those classes. Thanks for filling in the blanks ;)

[djelleleddine]

Thank you for the great and thorough explanations.
Though, in my opinion, the background music is disturbing.

[sephirothjc]

Well, so far my neural network works, I just copied all the code from the videos and tried every method in the neural network, I'm specially excited because I actually understand the whole thing.

[kkiller]

I did not realize this was an unfinished series.
*subscribes and awaits next video with bated breath*

[siddrameshwarkadagad]

Thank you so much for your videos. I loved watching them. It would be really helpful if you could please upload the codes that you have explained in these videos.

[skipy]

Love the video series. Both incredibly interesting and informative. Looking forward to future uploads!

[SaviorThree]

such phenomenal videos. thank you for this, keep up the great work. i

[drelthian]

Can't wait to get to pre-calculus next year... finally will be able to begin to understand this. :P

[LiftPizzas]

If your output landscape is smooth enough that this would be expected to work, then you can also use an evolutionary algorithm to explore it instead. What I mean is using mutation + selection, keep the best result, decrease mutation magnitude each time you improve. Once you've found a local minimum (you decide what constitutes one based on your needs, e.g. 20 or maybe 100 mutations with no improvement) you can store that as a best solution and increase the magnitude of mutation again, then either start mutating from your current best, or just select a brand new location. Keep track of your best solution, obviously. You decide how many iterations (local minimums found) are worth exploring, or that you can have it quit after some number of new attempts fail to improve on the old one. You might also find that many distant starting points settle on the same minimum, increasing your confidence about the smoothness of your landscape and quality of your solution. This still removes the "brute force" but allows the computer to take on the remaining grunt work.

[florianm]

Thanks a lot for this set of videos could you make an another set about convolutionnal neural network or other machine learning algorithms ?

[KID-jrib]

At 3:08, I'm confused by the syntax in computeGradients with the colon in front of the self.costFunctionPrime, because it's giving me an error and I can't figure it why, partly cause I'm also confused by the syntax, so I was wondering if someone could explain it to me? Thanks in advance!

[albertoherrerahp]

Thank you so much, you really help to learn this from 0. All tutorials at internet just explain the ecuations that they use to make the network but they never use it on a real case like you in python. I just have one doubt, is deep learning the same thing that you does in your vids? or deep learning means that the hidden layer is composed for 2 or more layers. Sorry for my bad english, but actually im from chile and my english is not so good. jaja Thanks again bro!

[nbynvne]

Numeric gradient = (f(x + epsilon) - f(x - epsilon)) / (2 * epsilon)

Given that epsilon = 1e-4, we can calculate the numeric gradient for x = 1.5 as follows:

f(x + epsilon) = (1.5 + 1e-4) ** 2 =
f(x - epsilon) = (1.5 - 1e-4) ** 2 =

Numeric gradient = - / (2 * 1e-4) = 0.02 / 2e-4 = 100

Therefore, the numeric gradient for the function f(x) = x**2 at x = 1.5 is 100.

I can't get the numeric gradient value to be could you advise?