Live Stream #118 - More Neural Networks

In this live stream, I make another attempt to discuss backpropagation with gradient descent. I also solve XOR with the new toy neural network library.

23:57 - Gradient Descent video
1:46:27 - 2nd Gradient Descent video
2:14:50 - XOR Coding Challenge

Schedule and topics:

Contact:

Links discussed in this video:
Stochastic Gradient Descent on Wikipedia

Videos mentioned:

[MultiLeandrini]

I'll be 100% honest, I tried to do the neuralnetwork this week before you could make this video but I was just doing what you did at 1:32:45 ahaha I knew what to do, I just couldn't decide how... ended up doing nothing
except making a new "model" of neuralnetwork creating a Perceptron class (in processing), that keeps track of it's weights, bias, output, and error. And making a NeuralNetwork class making a variable number of arrays of Perceptrons (each array beeing a layer in the network). I thought this way would make it really visual in my brain! Then we can easly code from here to make it appear on the screen.

[atrumluminarium]

For when it gets stuck, maybe "nudge" some of the weights by adding a small random number or reinitialise them completely? To test for these maybe look for where the error does a certain amount of iterations without moving but is not near zero. i.e. it fluctuates within an interval ±e1 but is more than ±e0 away from 0. The interval e0 could also be used to break out of the training loop if it achieves that accuracy before the iterations are complete as an optimisation option

[karthikd]

Video Request - Could you make a javascript video tutorial on performing Sentiment Analysis on Twitter and Reditt, using LSTM neural network and coming up with an optimal trading strategy (give buy/sell signals), validate the theory with past data and optimize it. . Please use opensource js libraries instead of P5.js, so that it's easier for us to replicate. Thank you so much!!

[KellyMurphy]

Something interesting happens if you go from 2 to 4 hidden nodes in the xor example. Each node in the hidden layer defines a line based on y=mx+b, so with 2 nodes, you get 2 parallel lines where output=1 and the middle undefined area approaches either 1 or 0 depending on the direction the lines are defined.

But if you change the hidden layer to 4 nodes, now you have 4 lines that have a chance of defineing a box. In this case, the four corners of the visualization are white and black. But the undefined center "square" where there is no training data, hovers around .5.

Based on the starting randomness there is no guarantee you end up with a box, the lines can still converge at the same solution that 2 hidden nodes comes up with.

While adding additional hidden nodes has the effect of more often defining the "box", the end result is the same as 4 hidden nodes when it does. This is because with 4 data points to train with, all lines must converge at 1 of 4 places.

Hope that makes sense. It's hard to describe this without visualizations.

[samuelmarques]

Very good! Finally, I was able to implement the gradient successfully on my neural network in python. Based on your channel, I see that javascript
is much more interactive than Python :)

[sherhy]

23:50 was i the only one cheering for him? :p

[nurjamil]

sir could you please do Neurovolution lesson? i need this for my undergraduate thesis xD

[KellyMurphy]

easyist to see below effect with 10000 training iterations per draw loop.

[ryaneakins]

Do Americans use the word "dithering" to describe what you were doing for the first couple of minutes here? It amused me to see the image dithering coding challenge on your desktop.

[realcygnus]

super cool deserve a glass of wine &/or a marijuana cigarette when you get home !