StatQuest: Principal Component Analysis (PCA), Step-by-Step

Principal Component Analysis, is one of the most useful data analysis and machine learning methods out there. It can be used to identify patterns in highly complex datasets and it can tell you what variables in your data are the most important. Lastly, it can tell you how accurate your new understanding of the data actually is.

In this video, I go one step at a time through PCA, and the method used to solve it, Singular Value Decomposition. I take it nice and slowly so that the simplicity of the method is revealed and clearly explained.

For a complete index of all the StatQuest videos, check out:

If you'd like to support StatQuest, please consider...

Buying The StatQuest Illustrated Guide to Machine Learning!!!

...or...

...a cool StatQuest t-shirt or sweatshirt:

...buying one or two of my songs (or go large and get a whole album!)

...or just donating to StatQuest!

Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:

0:00 Awesome song and introduction
0:30 Conceptual motivation for PCA
3:23 PCA worked out for 2-Dimensional data
5:03 Finding PC1
12:08 Singular vector/value, Eigenvector/value and loading scores defined
12:56 Finding PC2
14:14 Drawing the PCA graph
15:03 Calculating percent variation for each PC and scree plot
16:30 PCA worked out for 3-Dimensional data

#statquest #PCA #ML

[statquest]

NOTE 2: A lot of people ask about how, in 3-D, the 3rd PC can be perpendicular to both PC1 and PC2. Regardless of the number of dimensions, all principal components are perpendicular to each other. If that sounds insane, consider a 2-d graph, the x and y axes are perpendicular to each other. Now consider a 3-d graph, the x, y and z axes are all perpendicular to each other. Now consider a 4-d etc.
NOTE 3: A lot of people ask about the covariance matrix. There are two ways to do PCA: 1) The old way, which applies eigen-decomposition to the covariance matrix and 2) The new way, which applies singular value decomposition to the raw data. This video describes the new way, which is preferred because, from a computational stand point, it is more stable.
NOTE 4: A lot of people ask how fitting this line is different from Linear Regression. In Linear Regression we are trying to maintain a relationship between a value on the x-axis, and the value it would predict on the y-axis. In other words, the x-axis is used to predict values on the y-axis. This is why we use the vertical distance to measure error - because that tells us how far off our prediction is for the true value. In PCA, no such relationship exists, so we minimize the perpendicular distances between the data and the line.

[safaabdeljabbar]

For someone who is trying to be a data scientist, this channel is the best thing on the internet. You're better than any other teacher that I've ever had. THANK YOU.

[futurefriendly]

6 years into my phd and I finally understand how a PCA plot actually works. Thank you!

[rayskum]

Dr. Starmer - I really really wish I had you as my stats teacher during my student days. I can't put in words how much I appreciate your videos and how to go about explaining core concepts. Thank you very much!

[aizazkhan]

I can say without a shadow of doubt that you are one of the world's best teachers. Mr. Starmer I can never thank you enough.

[stephaniechen]

uni student day 14 of self-quarantine: the first thing to do in the morning is to watch StatQuest.
Thank you Josh. Your videos help me surviving thru my uni in the time of covid19

[kushnigam]

I am a data scientist and have performed PCA using advanced statistical softwares . I have even taken company sponsored expensive MVA courses. THIS is the best explanation of PCA I have seen and cleared my fundamental doubts and missing links. THANK YOU.

[ap]

This is the most helpful video on PCA I have ever seen. I'm a neuro PhD student and I was struggling to understand this concept for a year. This video is a life saver!

[rebecagarcia]

I had to pause and let you know, that this is gold! The way you simply describe terrifying names, it just makes it look so easy. Thank you so much

[manisharyal]

This is honestly the best explanation on PCA. connects lot of loose strings of ideas. thanks !!

[edchavez]

I'm studying for a final and this is way better explained than the entire semester.

[arianetalosig]

I love how you explained "linear combination is just basically this... no big deal" when my lecturer makes it such a BIG DEAL! Thank you!!!

[kushagrachaturvedy]

You are a real godsend. Teaching such advanced concepts in such a simple manner and that too in just 20 mins is exceptional

[josiahnielsen]

I'm in my first semester of grad school for stats.. and you are single-handedly going to save me! I already knew of all of the concepts of eigenvalues/eigenvectors, loadings, etc. but you summarized the 60 pages of theorems/proofs my prof had us read, and helped my intuition immensely. Please never stop making videos!

[kyoosikkim]

I just took some other online course for hours only to struggle... this video helped me understand the concept, which that course couldn't! Even in way shorter time! And even with the minimum animation! Always grateful

[himanshu]

This is such a complex topic and with the visualization you made it so simple, you really are magical in explaining the concepts, a million thanks for this Josh

[Oliver-]

Buddy you are an awesome guy. I wish one day I could give you a big hug of appreciation, but my honest thank you much has to be sufficient. I'd bet that there are people out there that would not be alive today without you. With your amount of views, there is probably at least one person who succeeded an important test that he would not have without you which may ended in a bad life. You're doing pure good with those videos.

[klapaucius-dkopf]

I've been watching quite a lot of your videos for the past few days to study for my finals and I'm so glad your videos exist! I'd be very desperate without them. Thank you so much for the amazing work! You have a natural talent of expressing complicated ideas so simply!

[takethegaussian]

I have seen no book explaining this topic better than you. Your skills and efforts are invaluable!

[zgqxyzs]

Very impressive! I searched a lot of websites and you are the only one that make the conceptions clear. Your graphs and explanations are vivid and easy to understand.