A Short Introduction to Entropy, Cross-Entropy and KL-Divergence

Entropy, Cross-Entropy and KL-Divergence are often used in Machine Learning, in particular for training classifiers. In this short video, you will understand where they come from and why we use them in ML.

Paper:

Errata:
* At 5:05, the sign is reversed on the second line, it should read: "Entropy = -0.35 log2(0.35) - ... - 0.01 log2(0.01) = 2.23 bits"
* At 8:43, the sum of predicted probabilities should always add up to 100%. Just pretend that I wrote, say, 23% instead of 30% for the Dog probability and everything's fine.

[revimfadli]

This feels like a 1.5-hour course conveyed in just 11 minutes, i wonder how much entropy it has :)

[jennyread]

Fantastic video, incredibly clear. Definitely going to subscribe!

I do have one suggestion. I think some people might struggle a little bit around 2m22s where you introduce the idea that if P(sun)=0.75 and P(rain)=0.25, then a forecast of rain reduces your uncertainty by a factor of 4. I think it's a little hard to see why at first. Sure, initially P(rain)=0.25 while after the forecast P(rain)=1, so it sounds reasonable that that would be a factor of 4. But your viewers might wonder why you can’t equally compute this as, initially P(sun)=0.75 while after the forecast P(sun)=0. That would give a factor of 0!

You could talk people through this a little more, e.g. say imagine the day is divided into 4 equally likely outcomes, 3 sunny and 1 rainy. Before, you were uncertain about which of the 4 options would happen but after a forecast of rain you know for sure it is the 1 rainy option – that’s a reduction by a factor of 4. However after a forecast of sun, you only know it is one of the 3 sunny options, so your uncertainty has gone down from 4 options to 3 – that’s a reduction by 4/3.

[ArxivInsights]

As a Machine Learning practitioner & Youtube vlogger, I find these videos incredibly valuable! If you want to freshen up on those so-often-needed theoretical concepts, your videos are much more efficient and clear than reading through several blogposts/papers. Thank you very much!!

[colletteloueva]

One of the most beautiful videos I've watched and understood a concept :')

[yb]

Thank you, I have always confused about these three concepts, you make these concepts really clear for me.

[agarwaengrc]

Haven't seen a better, clearer explanation of entropy and KL-Divergence, ever, and I've studied information theory before, in 2 courses and 3 books. Phenomenal, this should be made the standard intro for these concepts, in all university courses.

[xintongbian]

I've been googling KL Divergence for some time now without understanding anything... your video conveys that concept effortlessly. beautiful explanation

[Rafayak]

Finally, someone who understands, and doesn't just regurgitate the wikipedia page :) Thanks alot!

[AladinxGonca]

You are the most talented tutor I've ever seen

[metaprogand]

Phenomenal explanation of a seemingly esoteric concept into one that's simple & easy-to-understand. Great choice of examples too. Very information-dense yet super accessible for most people (I'd imagine).

[chenranxu]

Wow! It's just incredible to convey so much information while still keeping everything simple & well-explained, and within 10 min.

[gdbrukm]

I come to find Entorpy, but I received Entorpy, Cross-Enropy and KL-divergence. You are so generous!

[summary]

this is by far the best and most concise explanation on the fundamental concepts of information theory we need for machine learning..

[s.r]

Fantastic! This short video really explains the concept of entropy, cross-entropy, and KL-Divergence clearly, even if you know nothing about them before.
Thank you for the clear explaination!

[JakeMiller]

I always seem to come back to watch this video every 3-6 months, when I forget what KL Divergence is conceptually. It's a great video.

[khaledelsayed]

Very elegant indicating how cognizant the presenter is.

[paulmendoza]

I want to like this video 1000 times. To the point, no BS, clear, understandable.

[shiliseifeddine]

this is by far the best description of those 3 terms, can't be thankful enough

[SagarYadavIndia]

Beautiful short video, explaining the concept that is usually a 2 hour explanation in about 10 minutes.

[CowboyRocksteady]

i'm loving the slides and explaination. I noticed the name in the corner and thought, oh nice i know that name. then suddenly... It's the author of that huge book i love!