Factor Analysis and Probabilistic PCA

Factor Analysis and Probabilistic PCA are classic methods to capture how observations 'move together'.

SOCIAL MEDIA

SOURCES

[1] was my primary source since it provides the algorithm used in the Scikit Learn's Factor Analysis software (which is what I use). Since it walks through the derivation of the fitting procedure, it is quite technical. Ultimately, that level of detail came in handy for this video.

[2] and [4] were my go-to for Probabilistic PCA. A primary reason is Christopher Bishop is one of the originators of PPCA. That came with a lot of thoughtful motivation for the approach. The discussion there includes a lot advantages of PPCA over PCA.

[3] was my refresher on this subject when I first decided to this video. Like many of us, I'm a fan of Andrew Ng, so I was curious how he'd explain the subject. He emphasized that this model is particularly useful in high dimension-low data environments - something I forward in this video.

[5] is an excellent overview of FA and PPCA (as long as you're comfortable with linear algebra and probability). In fact, Kevin Murphy's entire book is like that for every subject and that's why it's my absolute favorite text.

---------------------------

[1] D. Barber, Bayesian Reasoning and Machine Learning, Cambridge University Press, 2012

[2] C. Bishop, Pattern Recognition and Machine Learning, Springer, 2006

[4] M. Tipping and C. Bishop, "Mixtures of Probabilistic Principal Component Analysers", MIT Press, 1999

[5] K. P. Murphy. Probabilistic Machine Learning (Second Edition), MIT Press, 2021

CONTENTS
0:00 Intro
0:21 The Problem Factor Analysis Solves
2:27 Factor Analysis Visually
5:52 The Factor Analysis Model
10:56 Fitting a Factor Analysis Model
14:13 Probabilistic PCA
15:43 Why is it Probabilistic "PCA"?
16:59 The Optimal Noise Variance

TOOLS

Social Media

[divine]

Thanks for covering this topic. I learned about and how to use FA and PCA in bootcamps but the way you dive into the internals is made so easily digestible.

[MikeOxmol_]

It's criminal that you don't have at least 50k subs. Please don't stop making videos, even though they don't have that many views right now, there are people like me who appreciate the videos very much. Certain topics can seem very daunting when you read about them, especially in such "dense" books as Bishop's PRML or Murphy's PML. However, if I start digging into a topic by watching your video and only then do I read the chapter, the ideas seem to connect more easily and I have to spend less time until it "clicks" if you know what I mean.

On another note, if you look for ideas for future vids (which I'm sure you already have plenty), Variational Inference would be a cool topic

[Nightfold]

This sheds some light into what I'm doing with PPCA but still I resent deeply my lack of formation in statistics during my degree.

[sasakevin]

The only reason that this guy's video didn't go viral is only 0.01% of the audience are interested in such complex statitics and formulas. But what he made are really awesome!

[mCoding]

Always love to hear your explanations!

[alan]

Thanks for the very clear explanation. I was doing my PhD under Chris Bishop when Bishop and Tipping were developing PPCA - good to get a refresher!

[mainakbiswas]

Had been finding this piece of information for quite a long time. I understood FA by sort of re-discovering it after seeing the sklearn documentation. From that point onward I wanted ro know why it related to pca. This have me the intuition and the resources ro look upto. ❤❤❤

[quitscheente]

Damn, I spend so much time going through 5 different books to understand PPCA and here you are, explaining it in an easy, comprehendable, visual manner. Love it. Thank you :)

[fenncc]

Great video, really informative, easy to understand, good production quality, and you've also got a great personality for these style of videos.

[enx]

True old school best techniques still in use them from 2004. They can save you as can build from nowhere amazing models

[Blahcub]

This was a super helpful video thank you so much. I love this material and find it super fun.

[jakubm]

great video, understandable explanations and cool format!

[EverlastingsSlave]

Man how good are your vidoes, i am amazed at perfection

[jonastjepkema]

Amazing! Hope your channel will eexplode soon!

[MrStphch]

Really really nice videos!! Love your way of explaining.

[wazirkahar]

Please please please keep doing this :)

[saeidhoseinipour]

Another nice video. Thanks 🙏
Please cover data science topics such as Clustering and Classification or applications like Textming, Recommender Systems, Image Processing and so on, with statistics perspective and linear algebra perspective.

[matej]

elite content, imho after the introduction I would love to see the content mainly, dunno if staying on screen makes the delivery better? whats the objective here ?

[taotaotan]

Hi DJ, awesome contents as always!!
I find I can follow your notations much better than textbook notations. At 8:12, I believe the matrix W is shared across all individuals, while z is specific to each sample. It makes intuitive sense to call matrix W common factors, and call z loadings. However, the textbook (Penn State Stat505 12/12.1) seems to call W (in their notation L) factor loadings, while calling z (in their notation f) common factors.
I am a little confused and I will appreciate it if you can take a look. Thank you again for the tutorial!

[michaelcatchen]

Around 10:35 you skip over the posterior inference of p(z_i | x_i, W, mu, psi) and that it is also a normal distribution because the normal is a conjugate prior for itself. Would love to see this covered in a separate video