Entropy (for data science) Clearly Explained!!!

Entropy is a fundamental concept in Data Science because it shows up all over the place - from Decision Trees, to similarity metrics, to state of the art dimension reduction algorithms. It's also surprisingly simple, but often poorly explained. Traditionally the equation is presented with the expectation that you memorize it without thoroughly understanding what it means and where it came from. This video takes a very different approach by showing you, step-by-step, where this simple equation comes from, making it easy to remember (and derive), understand and explain to your friends at parties.

For a complete index of all the StatQuest videos, check out:

If you'd like to support StatQuest, please consider...

Buying my book, The StatQuest Illustrated Guide to Machine Learning:

...or...

...a cool StatQuest t-shirt or sweatshirt:

...buying one or two of my songs (or go large and get a whole album!)

...or just donating to StatQuest!

Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:

0:00 Awesome song and introduction
1:28 Introduction to surprise
4:34 Equation for surprise
6:09 Calculating surprise for a series of events
9:35 Entropy defined for a coin
10:45 Entropy is the expected value of surprise
11:41 The entropy equation
13:01 Entropy in action!!!

#StatQuest #Entropy

[luischinchilla-garcia]

This is quite possibly the best explanation of entropy I've ever seen. This is even better than Shannon's own paper!

[felixvanderspek]

"Simplicity is the ultimate sophistication." - Leonardo da Vinci

Thanks for explaining in such simple terms!

[sade]

9 minutes of your video explained everything better than 2 hours of my professor giving a lecture... Thank you!!!

[eduardoh.m]

You, sir, you are the very first person to actually explain this subject and not just repeat some random definition without giving any thought to it. I'm amazed by the amount of people who confuse rambling on about the topic with actually explaining it. Thank you!

[deepakmehta]

Amazing video on Entropy Josh. Thank you. I am certainly more addicted to statquest than Netflix. I really liked the way you have introduced the notion of surprise, how you used it to pedagogically explain entropy. It is certainly now easy to think and remember the definition of entropy.

[CellRus]

Absolutely amazing. I always come back to your videos from time to time for simple (but absolutely useful) explanation of complicated concepts that I found in papers. They all have helped me a lot, and I feel I'm better at communicating these concepts to other researchers too.

[dvo]

best entropy explanation. I took a 500 level ML class last spring in my masters and this is better explanation than my prof(no disrespect to him, he is amazing too)

[saidisha]

One of the best explanations of entropy. I had been struggling for a while with this concept and there was no intuitive way I could understand and remember the formula so far, your video made it possible. Great video!

[nataliatenoriomaia]

Awesome as always, Josh! Thank you for continuing to share high quality content with us. You’re a very talented teacher. I wish you all the best!

[wolfgangi]

I freaking love these video, Josh has a gift to explain things so vividly

[danielpaul]

Starting the video with a message declaring that we can understand Entropy is the best starting line I have ever seen from any teacher in my life. Great work!!!

[loay]

Wow im sooo impressed. Frrr! It’s been a week trying to understand entropy and I rly thought I was never going to understand this bs. This video is arguably the best video on YouTube! Not only about entropy, but absolutely!! Thank you soo much

[shahedmahbub]

Thanks for all your efforts in creating a smart, funny and most importantly CLEAR explanation. This was awesome.

[hemanthhariharan]

I'm truly amazed by the power of simplicity and intuition. Hats off Josh!

[pietronickl]

Thanks for these super clear explanations – you really manage to break down complex concepts till they seem simple and (almost) intuitive. Also really appreciate the pace, ie the patience of going step-by-step and not making any crazy leaps 2 mins to the end 👏👏👏

[tranquilr]

Loved it. The best explanation I came across on Entropy. Thanks Josh!

[shamshersingh]

How can it be!! How can you simplify such complex topics into such simple explanations. Hats Off Man. I seriously wish if I could have had a Maths teacher like you back in school. I have become fan of your videos. Your videos are the first and last stop solution for all my doubts. Thanks Josh. You are a boon to learners like us. Impressed.

[sammitiyadav]

I follow most of your videos, not sure how I missed this gold! This is just the best entropy video I’ve ever seen.

[TuNguyen-oxlt]

this is definitely the most intuitive way to really gasp the ideal of entropy. you're just wonderful. Thank you soo much

[emmydistortion]

Awesome world-level teaching...
Thank you!