Visually Explained: Newton's Method in Optimization

We take a look at Newton's method, a powerful technique in Optimization. We explain the intuition behind it, and we list some of its pros and cons. No necessary background required beyond basic linear algebra and calculus.

00:00 Introduction
00:14 Unconstrained Optimization
01:18 Iterative Optimization
3:41 Numerical Example
6:00 Derivation of Newton's Method
7:48 Newton's Method for Solving Equations
8:33 The Good
9:31 The Bad
10:35 The Ugly

[quyenhuynh]

I'm a visual learner and this video is exactly what I'm looking for! Great content!

[jyothishkumar]

I'm here from yesterday's 3b1b video on Newton's method of finding roots, after wondering if there's any way to use it for minimizing a function. Mainly to see why we can't use it instead of Stochastic Gradiend Descent in Linear Regression. Turns out the Hessian of functions with many components can turn out to be large and computationally intensive, and also that if the second derivative is not a parabola, it can lead you far away from the minima. Still it was nice to see how the operation works in practice, and you mentioned the same points about Hessians too. Good job 😊👍

[rmrumman]

This is so high quality stuff! Thanks for the graphical explanation at the beginning!

[mattkriese]

Just finished calculus 1 and learning about Newton’s method brought me here. The visuals were fantastic and the explanation was clear. I’ll need to learn a lot more to grasp the entire concept, but it’s exciting to see topics taught like this for us visual learners.

Subbed 😁

[bradhatch]

What the what?! Even I understood this. Killer tutorial!

[aayushjariwala]

It's rare when less viewed video gives best explanation. Your presentations are almost like 3Blue1Brown or Khan academy! Don't know why this video has this less view!!

[SonLeTyP]

Hi Bachir, what an interesting series, very helpful. Cant wait to see next episode

[tuntstunt]

your videos are so good i wish they were a thing when I took my course on continuous optimization. my professor could never. i wish you would keep making them though!!!

[anoojjilladwar]

Hello Mr. Bachir El Khadir,
I recently came across your channel and was truly impressed by your videos and your clear explanations. I've just started working with AI and am also using the Manim library (created by Grant Sanderson) to make animated explanations.
I would really appreciate any advice you could offer, and I'm also curious to learn more about how you create your videos.

[SumitChauhan-vvix]

Brilliant visualization and explanation

[mlharville]

Loved this - very helpful! I knew this a long time ago and forgot much of it, so this was an excellent refresher, accessible to many. (And this is coming from a Stanford / Caltech grad working in computer vision, machine learning, and related things.)

[saqlainsajid]

This is brilliant thank you, hope you give us more visual insight into calculus related things

[shimuk]

FKKK !!!! I really wish I came across your video much before I took the painful ways to learn all this… definitely a big recommendation for all the people I know who just started with optimisation courses. Great work

[razmyhathy]

It is indeed a truly amazing explanation, and it helps me to understand Newton method visually.

[wenyunie]

Your explanation is awesome. Extension from root-finding scenario to minimum-point-finding problem was exactly my question.

[swazza]

Excellent video. I especially liked how you linked it back to the root finding version we learned in school. My one beef with this video is that that's an unfair depiction of Tuco.

[minoh]

Amazing explaination! This is very helful for understanding. Thanks a lot sir.

[sirelkir]

Another problem is for a negative curvature, the method climbs uphill. E.g. ML Loss functions tend to have a lot of saddle points, which attract the method, so gradient descent is used, because it can find the direction down from the saddle

[aniketbhandare]

It just needs more videos to get rocket growth !! Very Good Quality stuff ..

[samiissaadi]

Crystal clear explanation, thank you!