Coding Train Live 98: Starting Series on Neural Networks

In this live stream, I begin the long process of building a neural network library. I cover the concept of a "multi-layer perceptron" as well as linear algebra / matrix math.

Timestamps:
30:13 - Intro to Neural Networks
41:38 - Multilayered Perceptron Part 1
1:08:18 - Multilayered Perceptron Part 2
1:48:20 - Linear Algebra for Neural Networks Part 1
2:18:28 - Linear Algebra for Neural Networks Part 2
2:37:00 - Conclusion/Q&A

References:

Books discussed in this video:

My Videos mentioned in this video:

Help us caption & translate this video!

[rocktcatU]

This guy is the shining beacon of humanity.

[cabbage]

I think it's great you post the unedited version.

[betterculture]

Dan, I respect what you're doing and admire the energy you put into your contents -- it's a very colourful life you're living. You inspire me. Thank you so much!

Also, thanks for helping us discover JELITO :D

[grainfrizz]

At 56:00 I almost cried. It's like finding your soulmate. The sensation runs through my body. Thank you, Dan.

[afonsorafael]

I was having drum classes when this was on stream, but watching now, thanks for all of this

[tedchirvasiu]

2:17:00 the moment the legend of Jelito was born.

[Richard]

2:16:57 Awesome reaction! The console is king!

[MarcoAlessandrodeCampos]

Great video. Always nice to watch you!

[Ludifant]

Just a (very late) reaction to one of the questions from the chat: Yes! There is SOOO much value in rewriting existing libraries. It's the difference between thinking about a problem and solving it. If you don't it's like building an IKEA product. You get the product, but you don't learn anything about carpentry. Now if you just want to get the product, why bother with IKEA? Because it's cheaper. The same is true for programming only it's cheaper timewise. If you want to learn how to do carpentry or design your own programs: plough through. (And then start looking at how other people solved the same problem) Not before.. after...
Now if it's the right way to go in this video, which could have been structured and done in 15 minutes in stead of almost three hours, NO... But as a personal habit..? YES! I teach myself and even more than the subject matter, this is what I want them to learn. Because if they do, teaching becomes just pointing in the right direction.

[ShahriarAnwar]

someday I wish to meet you in person. you're such an inspiration:)

[ridilatrois]

Damn yes I'm intrested making a library!! It's been a very long time I keep repeating myself to try at least to do but i never comitted to it!

[luizfrg]

Dan, how about a programme for drawing electric field lines of two charges interacting? It'd be nice to see!

[culturapoliticaycomputador]

Hey Dan a video about Neuronal Networks great!! When are you going to program with a Qubit algorthm!!!

[martijnmulder]

What time does the Live Stream start? Eastern time, I can do the space time calculations to find my local time (Holland)

[martijnmulder]

The 'YouTube Chat', that you mention. Is it just in the Comments-section of the video. On the spot where I am typing this question right now?

[ashithshankar]

there is so much content I don't know where to begin

[kimoanhnguyenthi]

Could you make a video about how to make a button which is also a picture?

[bobsmithy]

Thanks for the vid! You saved me from making the same mistake. Also, is there any reason for making a function for scaling and adding matrices? I know multiplication of matrices is a must but why the other two?

[priswall]

when do you livestream? ive wanted to join one for a while

[periklisrips]

Please give me a moment to try and sort out the matrices issue. Since you go by the row-column convention the matrix will have to be like
|w11, w12, w13, ... |
|w21, w22, w23, ... |
|w31, w32, w33, ... |
| ... .... .... ... |,
where the first index has to indicate the ROW and the second one the COLUMN.
Now, in order to get the required result, what needs to be done is to multiply the matrices differently (it so happens that the order in dot product REALLY MATTERS). So for the dot product, every single row of the first matrix needs to be processed with each column of the second (remember, order matters).
This results to a m-by-n matrix (first) multiplied by a n-by-k matrix (second) to produce a m-by-k matrix.
So, here is how you can know whether you have it in the correct order or not.
Your weight matrix is a 2x2 matrix and your input is a 2x1 vector (but it could also be a 1x2 vector). If you want your dot product to be legit you have to put them either in this order (2x2) x (2x1), or in this order (1x2) x (2x2). The first case will produce a 2x1 vector and the second one a 1x2 vector. In the video you went with the first case and that seemed to be incorrect (it was ok for the video purposes but it caused all the misunderstanding). So let's see what will happen in the second case:
Note: This is the case of (1x2) x (2x2), so the input vector needs to be "horizontal"!
|x1, x2| x |w11, w12| = |x1*w11 + x2*w21, x1*w12 + x2*w22| = |h1, h2|
|w21, w22|
This turns out to be correct :)
Maybe a more practical method is better for understanding, but when it comes to modeling a problem that heavily depends on maths, theoretical consistency is a priority.
Hope this helps sort things out and I'm sorry for the long comment.
I really love your work though :)