22. Gradient Descent: Downhill to a Minimum

MIT 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, Spring 2018
Instructor: Gilbert Strang

Gradient descent is the most common optimization algorithm in deep learning and machine learning. It only takes into account the first derivative when performing updates on parameters - the stepwise process that moves downhill to reach a local minimum.

License: Creative Commons BY-NC-SA

[georgesadler]

Professor Strang thank you for a straight forward lecture on Gradient Descent: Downhill to a Minimum and its relationship with convex function. The examples are important for deep understanding of this topic in numerical linear algebra.

[tungohoang]

Very clear and natural to follow the lesson. Thank Professor Strang so much. Btw, his books are also very wonderful.

[Musabbir_Sakib]

Very natural way of teaching. Thank you sir

[satyamwarghat]

Wow the video quality is awesome
and lecture of Professor Gilbert Strang is the best

[mkelly]

Your lectures are a pleasure to watch (and learn from)!

[TheRsmits]

If in calc 1 they introduced the term argmin for the place where the minimum occurred there would be less confusion as students often mistake argmin for the actual min.

[martinspage]

his picture with grad(f) pointing up is a bit misleading around 9:00 I think. grad(f) is a vector in the x-y plane, pointing in the direction you should move in the x-y plane to maximize increase in f.

[naterojas]

Omg look at how clean those top boards are 🤩

[에헤헿-lv]

why in 42:25 insn't the gradient [x, by] since there is 1/2 multiplied at f?

[kirinkirin]

what a beautiful functions. that's why i love linear algebra.

[brainstormingsharing]

Absolutely well done and definitely keep it up!!! 👍👍👍👍👍👍

[samuelyeo]

How did he get all the equations of xk, yk and fk at 45:35? Specifically, how did he get (b-1)/(b+1) and vice versa? I shifted the equation to make xk+1 and yk+1 the subjects of the equations but instead I got xk+1 = xk (1-2sk), where xk = x0 = b.

[RC.]

Around 40:27. Does anybody know how to derive the result of reduction rate of m/M (the condition number)? Any tip or reference?

[HieuLe-unll]

I think the grad(f) at 16:00 should be 0.5(S+S tranpose)x-a, right? anyway, thank you for the amazing lecture!

[gopalkulkarni]

Isnt grad(f) supposed to be [x by] instead of [2x 2by]?

[finweman]

I am hoping for a discussion about conventions of derivatives. Much of the stuff I've seen would make the gradient a row vector, which leads to the derivatives being the transpose of what he shows. In his example, the derivative a'x is 'a' which is contrary to intuition from single variable calculus though he uses intuition for x'Sx.

[TheNeutralGuy]

For those who have not taken the previous 22 lecture, This lecture wont much help them.

[shenzheng]

In 26:51, the professor writes Gradient(f) = entries of X^-1. Do anyone know how to get that equation? Thanks!

[pnachtwey]

He is too long winded. Why not use a simple function of x, y. Find the derivatives and start dong a few iteration. Finally he gets to gradient descent. Gradient descent works but the are better algorithms. The line search idea is a good start. WTF is wrong with this guy? A simple python program or even excel would be much more meaningful. Thumbs down.

[John-wxzn]

This sounds like a bunch of non sense.