Neural Networks Demystified [Part 6: Training]

After all that work it's finally time to train our Neural Network. We'll use the BFGS numerical optimization algorithm and have a look at the results.

Supporting Code:

More on BFGS:

In this series, we will build and train a complete Artificial Neural Network in python. New videos every other friday.

Part 1: Data + Architecture
Part 2: Forward Propagation
Part 3: Gradient Descent
Part 4: Backpropagation
Part 5: Numerical Gradient Checking
Part 6: Training
Part 7: Overfitting, Testing, and Regularization

Follow me on Twitter for updates:
@stephencwelch

[TheStringKing]

these videos are too damn good ... i cant believe such perfect content exists on the internet

[afshinmeh]

I really like it when you say: "that's what we are going to do next time".

[christinehabib]

I'm taking a programming class and these videos have been an HUGE help

[mohamedayoob]

Went through the series recursively 7 times so far, and finally getting it. Perseverance really is the key. Thanks for such an amazing series. <3

[machtworobert]

Thanks to your lecture. Your illustration helps me understand deeper about ANN.
I am helping my mentor building a model. And I am sure I can help him achieve more now !

[rfew]

Very good series so far. I would love it if you also walked us through some of the current stuff: Deep Belief Networks (Deep Learning) and LSTM Recurrent Neural Networks.

[chelovekx]

Videos are very helpful at learning neural networks. Thank you so much!

[fannydagger]

Mate, honestly love these videos of yours. Cheers for all the effort!

[goodvc]

This video is very very helpful in understanding the ANN..
Thanks sincerely

[RapidDominance]

Im sticking in there. Amazing series xD

[spartakgrigoryev]

Hey there. Thank you for the videos. Genius presentation in such a simple and elegant wrapping!

[sarash]

These videos are just perfect. Thank you so much!

[pedrocrema]

Excellent videos! They are helping me a lot! Thanks for the great job!

[samperkings]

very helpful and clear. Thank you so much sir:)

[Lambertusjan]

Brain hurts at the end of video 5. Need to go back and repeat serveral times. Great stuff.
I did write a tiny ANN in python, but that performs less reliably thani would want. Possibly the over fitting issue.

[qiangzhang]

Why when I put the updated weight straightly to do forward propagation with test data, the prediction shows wrong results?

[brucinski]

I have enjoyed this series (minus the hand wavey linear algebra where a multiplcation is ambiguous where it shouldn't be and unexplained movement of matrices through entire expressions without justification) and have managed to follow it thus far. I am a PhD student in Nuclear Engineering and have 2 Bachelors in Nuke Eng + Math (numerical methods) from a good university (not bragging, I'm trying to explain a point to be introduced next). I find the "use code from a professional who did it" kinda silly when you are supposed to be teaching and explaining. We spent a long time implementing and checking gradients (2 entire episodes at least) only to be told "lol use this script" (I can't, as I wrote my code by hand in c++).

I really hope the instructor doesn't really think this material is approachable to high schoolers. Alongside that, I hope highschoolers don't watch this series and try to follow along piece by piece and feel inadequate bc they don't understand bachelor's + level concepts. Underselling the difficulty of the explanation of a concept may get more clicks and views and people (who just copied and pasted code or didn't even touch code) saying "omg so elegant you're better than all my professors" but it's hollow at times.

I'm not dogging this series, I have loved it and rewatched it until I mastered it tonight with the hope to apply some to my thesis, but it definitely wasn't basic high school calculus nor programming. I'm just super bummed that it didn't end with an explanation of the more advanced stuff (and the claim that this series wasn't about the advanced stuff is definitely untrue. a lot of these linear algebra concepts aren't touched at least till sophomore year in high school) and a step by step implementation of an expanded version of the initial concept.

I have watched all of your videos (and will continue to watch), so don't think I dislike the channel. I am just trying to provide constructive feedback :P

[Pucchini]

If you don't get yHat as close to y as in this video, make sure to initialize the weights with numpy.random.randn!
I stumbled because I used the numpy.random.rand function.
The difference is uniform (rand function) vs. normal (randn function) distribution.

[JRMGarcia]

Hi Stephen, excellent tutorial, the best. I'm trying to implement all of this in R and I have some doubts, could you help me, please? Why do you use setParams on both wrapper and callback functions? And, why do you return cost and grad in the wrapper function? Tks

[MultiSank]

Great series!

When I run the above code I get the error as "AttributeError: X" at the line self.N.setparams(_res.X)
Is there anything I am doing differently or need to do differently?
I also get a msg saying "Warning: Desired error not necessarily achieved due to precision loss."
I wonder if this is because of the above AttributeError?
All assistance appreciated!