Lecture 4: Optimization

Lecture 4 discusses optimization algorithms that are used to minimize loss functions discussed in the previous lecture. We introduce the core algorithm of gradient descent, and contrast numeric and analytic approaches to computing gradients. We discuss extensions to the basic gradient descent algorithm including stochastic gradient descent (SGD) and momentum. We also discuss more advanced first-order optimization algorithms such as AdaGrad, RMSProp, and Adam, and briefly discuss second-order optimization.

_________________________________________________________________________________________________

Computer Vision has become ubiquitous in our society, with applications in search, image understanding, apps, mapping, medicine, drones, and self-driving cars. Core to many of these applications are visual recognition tasks such as image classification and object detection. Recent developments in neural network approaches have greatly advanced the performance of these state-of-the-art visual recognition systems. This course is a deep dive into details of neural-network based deep learning methods for computer vision. During this course, students will learn to implement, train and debug their own neural networks and gain a detailed understanding of cutting-edge research in computer vision. We will cover learning algorithms, neural network architectures, and practical engineering tricks for training and fine-tuning networks for visual recognition tasks.

[jaeen]

As a CS grad student myself, I've sat through many lectures. This professor is really, really good.

[muhammetcavus]

great lecture again, even though I did not understand anything.

[mohammadvahidi]

love these lectures
thanks man, you are an amazing professor.

[baskaisimkalmamisti]

Animations:
33:41 SGD
38:06 SGD + Momentum
45:05 Nesterov
50:23 RMSprop
55:27 Adam

[VikasKM]

This is best lecture on optimizations, very clearly explanations, SGD, SGD+Momentum, Adagrad, RMSProp and Adam.
If you have doubt i would request you to watch this with Andrew Ng Deep Learning Specialization Lectures to get a clear picture around optimization.

[alexanderfrei]

Absolutely brilliant !!
Since you have went through the details (instead of jumping over them) I finally understood how derivative is taken in ML – Kudos!! 😊

[riadelectro]

Thanks a lot professor, i will be grateful if you add a video course for proximal gradient methods.

[changjuanjing]

Great lecture, I came back to watch again

[BoTian]

Great lecture, Justin. Wonder what you think about MADGRAD?

[anishmanandhar]

25:06 Isnt the loss in SGD computed for an example, what differences would SGD have over Minibatch gradient descent then?

[syed]

please fix voice in next recordings...