Momentum-based gradient descent from scratch: optimization | Foundations for ML [Lecture 24]

Momentum Gradient Descent: A Smarter Way to Optimize Machine Learning Models

Optimization is key in machine learning, and while Gradient Descent lays the foundation, it often struggles with inefficiencies like slow convergence and oscillations. Enter Momentum Gradient Descent—an upgrade that accelerates learning and smooths optimization. Let’s break it down.

The Problem with Standard Gradient Descent
Gradient Descent updates model parameters by taking small steps in the direction of the negative gradient of the loss function. While effective, it has a few challenges:

Slow Convergence: In flat regions (e.g., plateaus), progress is sluggish.
Oscillations in Narrow Valleys: The updates zigzag across steep areas, delaying progress.
Learning Rate Sensitivity: A step that’s too small takes forever, and one that’s too large causes divergence.

How Momentum Helps
Momentum Gradient Descent adds a “velocity” term that builds on past gradients, allowing the algorithm to keep moving in the right direction even in tricky terrains. Think of it like pushing a sled downhill—momentum helps you glide over bumps smoothly instead of getting stuck.

Here’s how it works:

Velocity Update:
The velocity accumulates past gradients:
velocity = (momentum_factor × previous_velocity) - (learning_rate × current_gradient)

Parameter Update:
The parameters are updated using the velocity:
new_parameters = old_parameters + velocity

The momentum factor (often set to 0.9) controls how much influence past gradients have on the current step.

Benefits of Momentum
Faster Convergence: Momentum builds speed in consistent directions, reducing the time to reach the optimal solution.
Reduced Oscillations: In narrow valleys, the velocity helps smooth out the zigzag pattern of standard Gradient Descent.
Stable Updates: It’s less sensitive to noisy gradients and allows for slightly larger learning rates.

Where Momentum Shines
Momentum Gradient Descent is particularly effective for:

Deep learning: Training neural networks with complex, bumpy loss surfaces.
High-dimensional problems: Where steep gradients in some directions cause oscillations, and flat regions slow progress.

The Takeaway
Momentum Gradient Descent is a smart extension of standard Gradient Descent. By incorporating gradient history, it smooths out noisy updates, accelerates convergence, and handles challenging optimization landscapes with ease.

If Gradient Descent is like carefully stepping downhill, Momentum Gradient Descent is like rolling a ball—it builds speed while staying on track. For machine learning practitioners, it’s a reliable way to optimize models efficiently.

What’s your experience with Momentum Gradient Descent? Are there other optimizers you rely on? Let’s share insights in the comments!

[ShreyJha]

Please correct me if I misunderstood, at 16:00 minutes, the velocity should not be going up, but slowing down as the reminisce of previous. Unlike vanilla Gd, where there is no velocity. Or am I mistaken.

[rahul_IIT-Ghy]

Can you please cover noise extraction from any high frequency data, using by like cloud computing....

[najamulhassan]

Hi, would you share the code, so we can play around. Thank you for your efforts and time .