What is Machine Learning? Part 3/5: Decision Tree Algorithm explained - Entropy

Показать описание

In this video, we are going to talk about an extremely important part of the decision tree algorithm, namely the metric that tells us where to best split the data. Or in other words, the metric that tells us what questions to ask. And the particular metric that we are going to look at is called entropy.

Edit: in the formula of the overall entropy at 16:35, "Entropy" should also have a subscript "j"

Links:

Рекомендации по теме

Комментарии

Clarification on Information Gain vs Overall Entropy (17:45)

The formula for Information Gain is basically like this:
Information Gain = Entropy before split – weighted Entropy after split

Or in other words:
Information Gain = Entropy before split – Overall Entropy

So, to determine the Entropy before the split, we need to calculate the following:
Entropy before split = 42/130 * (-log2*42/130) + 42/130 * (-log2*42/130) + 46/130 * (-log2*46/130) = 1.584

So, the Information Gain for split 1 is:
Information Gain = 1.584 – 0.646 = 0.938

And the Information Gain for split 2 is:
Information Gain = 1.584 – 0.804 = 0.780

So, split 1 results in a higher Information Gain and we would chose it over split 2. Therefore, we get the same result compared to the video, where we just used Overall Entropy.

The reason I decided to just use Overall Entropy and not Information Gain is because they are essentially the same. With Overall Entropy you focus on the fact that the entropy decreases from 1.584 to 0.646 after the split. And with Information Gain you focus on the fact that the entropy of 1.584 decreases by 0.646 to get an Information Gain of 0.938 after the split.

In my opinion, using Overall Entropy is simply more intuitive. Additionally, it requires one less calculational step.

SebastianMantey

I can't thank you enough for sharing such valuable information !!

hazemahmed

Excellent explanations. want to see more videos from you.

rajatpati

Thank you. Very clear explanation and visualisations.
Would be great if you gave some words on how these are used for regression.

SuperIdo

What is Machine Learning? Part 3/5: Decision Tree Algorithm explained - Entropy

Machine Learning | What Is Machine Learning? | Introduction To Machine Learning | 2024 | Simplilearn

What is Machine Learning?

Intro to Machine Learning (ML Zero to Hero - Part 1)

But what is a neural network? | Chapter 1, Deep learning

Machine Learning for Everybody – Full Course

Supervised vs Unsupervised vs Reinforcement Learning | Machine Learning Tutorial | Simplilearn

Machine Learning Tutorial Python -1: What is Machine Learning?

Introduction to ML and AI - MFML Part 1

Deep Learning Chapter 14 part 1 : Computer Vision Using Convolutional Neural Networks

What is Machine Learning? | Machine Learning Basics | Machine Learning Tutorial | Edureka

What is a Machine Learning Engineer

What Is AI? | Artificial Intelligence | What is Artificial Intelligence? | AI In 5 Mins |Simplilearn

What Is Machine Learning? | What Is Machine Learning And How Does It Work? | Simplilearn

Machine Learning Tutorial Part - 1 | Machine Learning Tutorial For Beginners Part - 1 | Simplilearn

Learning to See [Part 4: Machine Learning]

Learning Machine Learning has never been easier #shorts #machinelearning #statistics #datascience

Deep Learning vs. Machine Learning, which is better?

Computer Scientist Explains Machine Learning in 5 Levels of Difficulty | WIRED

How to Get Started with Machine Learning & AI

Artificial Intelligence (AI) vs Machine Learning vs Deep Learning vs Data Science

Neural Network In 5 Minutes | What Is A Neural Network? | How Neural Networks Work | Simplilearn

Machine Learning Foundations: Ep #1 - What is ML?

Is this still the best book on Machine Learning?

Machine Learning Full Course - Learn Machine Learning 10 Hours | Machine Learning Tutorial | Edureka