CS231n Winter 2016: Lecture 6: Neural Networks Part 3 / Intro to ConvNets

Stanford Winter Quarter 2016 class: CS231n: Convolutional Neural Networks for Visual Recognition. Lecture 6.

Get in touch on Twitter @cs231n, or on Reddit /r/cs231n.

[citiblocsMaster]

1:04:50 "This might be useful in self driving cars". One year later, head of AI at Tesla.

[irtazaa]

Did cs231n in 2015. great to see the videos released to public now.
Good Job Stanford!

[caner]

That project report being mentioned at 27:13 had been accepted to one of the ICLR workshops on the following year and has over 500 citations up until now. Impressive stuff.

[vivekloganathan]

For someone curious like me @ 43:34
(Someone's siri mistakenly tried to recognize.. and said this..)
Siri: "I am not sure what you said"

[leixun]

*My takeaways:*
1. Parameter updates: optimizers, such as momentum, Nesterov momentum, AdaGrad, RMSProp, Adam 3:53
2. Learning rate 28:20
3. 2nd order optimizers 30:53
4. Evaluation: model ensembles 36:19
5. Regularization: dropout 38:25
6. Gradient checking 56:55
7. Convolutional Neural Networks 57:35

[champnaman]

@15:10, Andrej says that according to recent work, local minimums' are not a problem for large networks. Could anyone point me to these papers? I am interested to read these results.

[boringmanager]

Some guy playing dota vs neural network - over million views.
A genius explaining how to build a neural network - 40k views.

[tthtlc]

Just found that the slides at Stanford website is updated with the 2017 slides + videos. Is there any way to get the original 2016 slides? The lectures are as classic as those of Andrew Ng.

[ThienPham-hvkx]

Summary : start from a simple gradient decent : x += - learning_rate * dx, but when we applied this gradient decent in a big dataset, it will take a long time to calculate derivative for each data point (neural unit). So we will use Stochastic Gradient Descent (SGD) to randomly choose a neural unit in a layer instead of whole units. It will reduce the time to calculate derivative.
SGD 's still slow because it jitter on the way of convergence because of random. Then, we have other method to help converge faster: SGD + Momentum, Adagrad, RMSProp, Adam . Each of them will have learning rate. We should find the ideal learning rate for each of data set. For ex: default learning rate of Adam = 0.001 in Keras.
We can use dropout to prevent overfitting.
To prevent overfitting, we have 3 ways: increase dataset, simplify network (dropout, reduce number of layer), preprocessing data (data augmentation)

[twentyeightO]

This is helping me quite a lot, thanks!!!

[mostinho]

Intro to convnets is all history, skip to next lecture for convnets

[ArdianUmam]

In 9:12, the SGD refers to literally SGD (train with ONLY 1 example randomly) or refers to mini-batch? Because in the web lecture notes, it states that term SGD often enough to be used for "mini-batch" actually (not literally SGD using ONLy 1 example).

[sokhibtukhtaev]

at 44:06, Is he saying that dropout is applied on different neurons in each epoch? Say, I have x -10-10 - y network (x -input, y- output, 10s - hidden layers). In one epoch (that's forward prop + backprop) dropout is applied to, say, 3rd, 5th, 9th neurons of first hidden layer and 2nd, 5th, 8th neurons of second hidden layer, in the second epoch dropout is applied to 5th, 6th, 10th neurons of first layer and 1st, 7th, 10th neurons of second hidden layer. Does it mean we kind of have as many models as our epoch? Can someone clear this up for me?

[omeryalcn]

there is a one issue . We don't suggest normalized data when it is a image. but when we use batch normalization , we normalize data. Is that a problem ?

[qwerty-tfzn]

It's getting exponentially difficult

[randywelt]

8:50 noisy signal = how about usage of kalman filter??

[nguyenthanhdat]

The best AI course I have ever taken. Thank you, Andrej!!!

[jiexc]

What happended on 43:38, what's so funny?

[WahranRai]

33:12 L-BFGS not be confused with LBGT

[nikolahuang]

The big idea of momentum update method is smart! But it is obvious that this updating method is an interim method.