CS231n Winter 2016: Lecture 3: Linear Classification 2, Optimization

Stanford Winter Quarter 2016 class: CS231n: Convolutional Neural Networks for Visual Recognition. Lecture 3.
Get in touch on Twitter @cs231n, or on Reddit /r/cs231n.

[KhueLe]

A big thank to Andrej and the teaching staff for uploading the videos. They are of high quality and really make the class more interesting to follow.

[leixun]

*My takeaways:*
1.Multiclass SVM loss 4:15
2. Regularization 20:04: e.g. L2, L1
3. Softmax classifier 29:40
4. Optimization 50:10: gradient descent
4.1 Mini-batch gradient descent 1:00:40
4.2 Optimizers 1:05:00: SGD, momentum, Adagrad, RMSProp, Adam

[NehadHirmiz]

These lectures are amazing. I still go back to review these every few years. Thank you sir.

[sezaiburakkantarci]

You are one of the best Andrej. You make learning so fun, with moments like 27:45 😄

Forever grateful.

[njordeggen]

The best thing about the internet is access to great information and education like this series. Thank you Andrej and everyone involved!

[bayesianlee]

Not only his lecture itself but his intelligence level and interpretation level on DL is quite interesting. Thanks for sharing and grateful for classes.

[Shkencetari]

Thank you very much for this great lecture.

Just to help others out, optimization starts at 50:11

[geoffreyanderson]

Instructor's losses math is messed up at 8m31s. For frog colum the loss should be 12.9 but instructor says total loss of frog column is 10.9:

In [10]: max(0, 2.2 - (-3.1)+1) # cat row frog column

Out[10]:

Etc.

[ralphblanes]

Andrej's a great teacher! Keep up the good work!

[joehan]

On slide 11, shouldn't max(0, 5.3) + max(0, 5.6) be max(0, 6.3) + max(0, 6.6)?

[XTLi-xbiv]

Thanks for uploading the video online, it really helps!

[vaibhavravichandran]

Thank you for the lecture series. They've helped me understand NNs much much better.

[zahrasoltani]

Thanks Andrej for the nice lecture. Just one point in the early minutes o the lecture, the car score is good but it is mentioned as bad. the score is high which means high probability so it is not a bad score.

[pauldacus]

17:00 starts a good discussion of regularization...

[deepuraveendran]

Loss Function of Frog at 8.36 is 12.9 not 10.9, forget to add +1

[drdrsh]

There is a very interesting artifact around Andrej Head, the symbols on the white board seem to jiggle around as his head moves. Warping the space-time somehow?

[MauriceMauser]

[33.17] (-)0.84 is log base 10. assume, we want the natural log: - ln(0.13) = 2.04 ?

[Siwon-vvmi]

At 39:00, what did he mean by jiggling the scores?

[ThienPham-hvkx]

Summary: Loss function let us know how well the randomized weight we choose going ? (Bad, good, is going bad, is going good). To minimize the loss, we will try to press it to ZERO as much as possible in whole dataset (maybe million samples), look like a blind person 👨‍🦯 try going down hill ⛰️ and on each step he make sure that he is on right way of going down.

[ShangDaili]

I looked at the notes, and I still don't understand why analytical gradient is prone to errors. Can somebody explain? Thanks.