Intro to Gradient Descent || Optimizing High-Dimensional Equations

How can we find maximums and minimums for complicated functions with an enormous number of variables like you might get when studying neural networks or machine learning? In this video we are going to talk about a topic from nonlinear optimization called Gradient Descent (or Gradient Ascent if you want maximums) where you step-by-step approach an extremum by stepping in the direction of the gradient vector. We're going to see the basic algorithm, see some common pitfalls, and then upgrade it using a method called line searches to improve the efficiency.

Check out my MATH MERCH line in collaboration with Beautiful Equations

COURSE PLAYLISTS:

OTHER PLAYLISTS:
► Learning Math Series
►Cool Math Series:

BECOME A MEMBER:

MATH BOOKS I LOVE (affilliate link):

SOCIALS:

[spongebobinator]

It's amazing how well the timing of this video came out. I sat down to write an article on neural networks, went on YouTube to play some background music, and had one of my favorite educators show up on my home screen with this extremely relevant topic. Awesome!

[ProbablyApproximatelyCorrect]

Nice video! For being so simple, it is fascinating how well gradient descent and its variants work. Like you mention, it is not at all guaranteed to find a good minimizer for non-convex functions in low dimension, but in higher dimensions, things just magically seem to work out, both numerically and (recently) mathematically. There's so much about high-dimensional objects that isn't really captured by our low-dimensional intuition, which unfortunately is quite limited. I recently saw a quote from Geoff Hinton that said:
"To deal with a 14-dimensional space, visualize a 3-D space and say 'fourteen' to yourself very loudly."

[vincentalegrete]

Best math channel on YouTube. The way you distill things down to intuitive graphics really helped me get more out of Calc III last semester.

[jinc]

Hello Dr. Bazett, I have been watching a lot of your videos during my current PhD process. I just want to say that you made hard concept visual and more intuitive. it has been a great help to me! Please continue making more! I love to see you explaining some convergence analysis!

[fatianass]

Optimizing functions ❌️
Something we've heard about it and didn't give it enough time in that 30 hours machine learning course ✅️
thanks for the vid!

[mohammedmhilal]

I literally search Gradient Descent Trefor bazett two days ago... This is destiny....

[judedavis]

Now this is amazing! Please continue the ML math stuff!!

[pierreretief]

This video has been very interesting. I have really enjoyed and benifitted from your videos, especially the ones on vector calculus. When I did my engineering degree ten years ago, either I wasn't aware of these types of videos, or they didn't exist. I am currently studying a physics degree remotely and these videos have been IMMENSELY helpful. You and prof Leonard and many others are changing lives. I understand concepts now that I didn't even realize I didnt understand the first time around. The graphics and your teaching style - awesome. Thank you very very much.

[babybeel]

Hello, that was perhaps one of the most beautiful explanation and visualisation of the gradient I've come across, I could not be happier after watching this video, your excitement is what mathematics is all about, discovering beautiful things. Thank you so much !

[teikono]

Just want to say this channel is fantastic, its very enjoyable listening to you talk about these topics.

[Heisenberg]

dude, I am studying cs and getting an expert in ai, but I found your account because of latex. Now this gets recommended to me. The recommendation algorithm predicting at it's finest ;D

[kaygeea]

Great video!! Please tell Brilliant that our mobile app needs a dark mode ASAP. Love their stuff

[erkangwang]

This is so much better than a lot of text books I read

[bhavesh.adhikari]

I was just thinking about this topic since several days. now the video is here. thanks, Prof. you have helped me a lot

[TimLeung]

When I did my PhD in aerodynamic shape optimization for airplanes, we would have hundreds or thousands of variables to define the shape of the wing, and the “function” that we were minimizing (e.g. drag at a fixed lift) cannot be defined analytically (it required a full flow analysis), there were a lot of research done in finding the gradient direction. Also, while we would start with the steepest descent direction, we would also build an approximate Hessian matrix that gives us a progressively clearer picture of the function space after every step. The most efficient step sizes that we would use were quite different from the steepest descent line search.

[Kokurorokuko]

2:52 If we want to maximize, shouldn't we add the gradient? Of course gamma can be negative but just conceptually?

[egycg]

Unbelievable explanation as usual, prof.
Thanks

[geraldsnodd]

This year i will be going to college once i am finished with my highschool exams and entrance exams. Can't wait to explore all of your content in college.

[reckless]

i'm in the 9th grade, I have understood 1/27 of what you're talking about. Ig i have to watch this 27 more times

[tooooomboh]

Amazing visualizations, really well done!!!