Regularization Part 1: Ridge (L2) Regression

Ridge Regression is a neat little way to ensure you don't overfit your training data - essentially, you are desensitizing your model to the training data. It can also help you solve unsolvable equations, and if that isn't bad to the bone, I don't know what is.

This StatQuest follows up on the StatQuests on:
Bias and Variance

Linear Models Part 1: Linear Regression

Linear Models Part 1.5: Multiple Regression

Linear Models Part 2: t-Tests and ANOVA

Linear Models Part 3: Design Matrices

Cross Validation:

For a complete index of all the StatQuest videos, check out:

If you'd like to support StatQuest, please consider...

Buying The StatQuest Illustrated Guide to Machine Learning!!!

...or...

...a cool StatQuest t-shirt or sweatshirt:
...buying one or two of my songs (or go large and get a whole album!)

...or just donating to StatQuest!

Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:

0:00 Awesome song and introduction
1:25 Ridge Regression main ideas
4:15 Ridge Regression details
10:21 Ridge Regression for discrete variables
13:24 Ridge Regression for Logistic Regression
14:12 Ridge Regression for fancy models
15:34 Ridge Regression when you don't have much data
19:15 Summary of concepts

Correction:
13:39 I meant to say "Negative Log-Likelihood" instead of "Likelihood".

#statquest #regularization

[statquest]

Correction:
13:39 I meant to put "Negative Log-Likelihood" instead of "Likelihood".

A lot of people ask about 15:34 and how we are supposed to do Cross Validation with only one data point. At this point I was just trying to keep the example simple and if, in practice, you don't have enough data for cross validation then you can't fit a line with ridge regression. However, much more common is that you might have 500 variables but only 400 observations - in this case you have enough data for cross validation and can fit a line with Ridge Regression, but since there are more variables than observations, you can't do ordinary least squares.
ALSO, a lot of people ask why can't lambda by negative. Remember, the goal of lambda is not to give us the optimal fit, but to prevent overfitting. If a positive value for lambda does not improve the situation, then the optimal value for lambda (discovered via cross validation) will be 0, and the line will fit no worse than the Ordinary Least Squares Line.

[scubashar]

I am a machine learning engineer at a large, global tech company with a decade of experience in industry and a computer science graduate student. Your channel has helped me immensely in learning new concepts for work and job interviews, and your videos are so enjoyable to watch. They make learning feel effortless! Thank you so much!!

[PolitePolice]

This channel is by far the best at explaining mathematical concepts related to machine learning. I'm in a machine learning class at my university and go to every class lecture. I leave not having understood an hour and fifteen minutes of lecture. I immediately pull up this channel and watch a video on the same concept and "BAM". It makes sense.

[ryzary]

After watching dozens of StatQuest videos, I finally know when to say 'BAM!'

[ardakosar]

Explaining things at this complexity at this level of simplicity is a real skill! Awesome channel!

[lucaspenna]

Professors in general teach Ridge Regression with many complicated equations and notations. You made this topic very clear and easy to understand. Thank u very much again.

[Nicole-sezj]

I've spent so much time trying to read and understand what EXACTLY is ridge regression. This video made it much easier to understand. Thank you so much for simplifying this complex concept!

[EvaPev]

I have no words to express how good this lecture is.

[TheGoldenFluzzleBuff]

I have a big data economics exam tomorrow and you literally just saved my life. I don't always understand what my professor is trying to explain, but you did it super clearly. Actual life saver

[NaggieNag]

I don't know how my stat teacher can make something this easy to understand that complicated. Everytime I can't understand what he's talking about in the class I know that I have to turn to StatQuest. Thank you for what you're doing.

[elliotyip]

The way you go through the logic step by step makes you a good teacher. In many of my research occasions they just say "adjust your alpha higher or lower until you don't overfit / underfit" but I don't even know what am I looking at. Bless you.

[JT]

Josh, I have been practicing data science since last 4 years and have used Ridge regression as well. But now I am feeling embarrassed after watching this explanation because before the video I only had half baked knowledge. You deserve a lot of accolades my friend :)

[pritamck]

Only Statquest can make someone emotional while learning statistics. The ease with which the concepts are flowing flawlessly into my brain makesme teary. Thank you so much 🥺❣

[anamfatima]

I came to know about this channel 2 hours ago. Simple and Outstanding explanation. My aim is to watch each and every video.
Loving your style of teaching.
From India.

[theyearpuzzle]

I am brand-new to statistics, and I'm in school to be a data scientist. so many times, I lose the plot watching lectures from my professors who have the Curse of Knowledge. I end up spending hours watching your videos and they help so much, I just don't even have words! I've recommended your channel to all my classmates--and I mentioned it so much, my professor is considering adding your channel to recommended materials for next semester! you are a shining light of joy in a jargon-filled sea of confusion.

[charissapoh]

your explanations are insane... they're so easy to understand and literally capture the essence of the topic without being overly complicated! i've bingewatched so many of your videos ever since chancing upon your channel last night - i specially love the little jingles you add in at the start of your videos, they really add such a fun and personal touch~ thank you so so soo much, your channel has really helped me immensely!!!

[SomeOfOthers]

I've taken 4 machine learning courses and always wondered what ridge regression was, because I've heard it several times, but I was never taught it. I never realized it was just adding the regularization parameter! Awesome! Thank you so much.

[andersonarroyo]

This is my first video and I am so impressed by how you explain things!!! It is like my buddy from college will explain it to me in plain words. You rock StatQuest, I am a follower from now on!! Thank you

[SpL-muzu]

YOU ARE THOUSANDS OF TIMES BETTER THAN MY PROF...CLEAR & SIMPLE.

[DragomirJtac]

Incredibly clear explanation. I'm using your Machine Learning videos to study for my midterm for sure. It's so nice to know that these concepts aren't above my head after all.