Effect Size

This video explains what effect size means when reading educational research and was designed for use by RSE-TASC members in New York. The .3 mentioned was our bottom limit for deciding if a practice was effective. John Hattie (Visible Learning) states that .4 is a better floor and we have changed our practice to reflect that.

[johnboniello]

After reading and answering a couple of comments with similar questions I decided to post an explanation that I left out in the video of how to find the amount of "spots" the student moves. I hope this helps anyone with the same question. If you are still confused or have follow ups please comment!

Thanks to those that posted the questions in the first place! tl;dr, it comes down to using ES as a z-score in a standard normal distribution which tells us how many standard deviations the student would have moved when compared with the control. This then gives us a percentile, which allows us to calculate the rank.

This might take a bit to explain, and I'll be using some of the information form the paper I cited for this information. I hope this will answer your question, if not please follow up.
Basically, an effect size is simple way to evaluate the effectiveness of an intervention by comparing two groups, one control and one experimental. "Effect size is exactly equivalent to a 'Z-score' of a standard distribution" (Coe).

When we look at our list of students ranked by performance in two classrooms, like in the video, we see that they are about the same and John is the average student in one of the classes. After giving the intervention to John's class and leaving the other class alone we can compare scores. John was the average student, he was in the 50th percentile before the experiment.

After the experiment we compare the two groups and get an effect size of 0 lets say, which means the intervention had no effect. John is still in the 50th percentile because there was no movement. If we looked at the two normal curves of the two classes they would overlap. If there was an effect of .5 that would be a movement of .5 standard deviations in a positive direction (to the right) which would mean John, originally in the 50th percentile compared to the other group, would move to the 69th percentile when compared to the control group. In a class of 25 that would put him at number 7. If it was an effect of 1 that would be 1 standard deviation to the right and would be in the 84th percentile, or in a class of 25 number 4. If the group size was larger, lets say 100, the ranks would be different as they would correlate to the percentile (84th percentile means they scored higher than 84% of the class). Page 4 of the paper cited has a great table that shows this in action.

Also, this works in the negative direction, so if the average student in the 50th percentile was actually hurt by an intervention and had worse outcomes (which does happen!) the rank would drop. An ES of -.5 would mean the student would be .5 standard deviations to the left (negative direction) and would now be in the 31st percentile, instead of scoring higher than 50% of the class he now only scores better than 31% of the class.

This is how we can assign position based on effect size, we get the effect size, which correlates to Z-score, which tells us how many standard deviations the experimental group moved, which leads us to a conclusion of percentile. From there we take our class size and figure out what it means to be in the nth percentile. I really hope this long-winded explanation helps!

[tralala]

This did more for me than an entire semester's worth of statistics classes. Thx bro

[madisonhaug]

I just wanted to say thank you so much for this video. I am in my first year of grad school taking statistics and Small Case Design and I have been struggling with understanding effect sizes. After watching your video, it makes so much more sense! You are amazing!

[nickjpriore]

Thank you, John, You certainly are helping many people to the top of their class!

[Anonymous-gmgm]

You've managed what several university lecturers have failed at - getting me to understand this part of stats. Gutted you haven't done more videos about stats!

[pueraeternus]

Thank you so much! It's helping me in my first semester of grad school.

[powerpointpro]

so far the best explanation out there!

[Come-and-take-it]

by far the BEST explanation I found on youtube. thanks

[deborahgracedabuet]

The only thing I'm thinking right now is.. What in the world is this "intervention" that made John top his 24 peers?! Lol.

[interestingvideoshaha]

love john!he makes understanding research papers a lot easier😁thanks heaps!

[warnacokelat]

Helped me wrap my head around this stuff. Hopefully I can submit the report on time..

[alistaircollins]

Amazing video! The explanation was simply fantastic. I believe you only have less than 400 likes because of the non-mainstream subject but the qualify is there. The graphics were very basic , but hey man, they get the point across.

[KhalidMahida]

I am new to the statistics, excellent explanation of the phenomenon, Effect size

[OluwayomiAdeyemi]

Awesome! Well explained concept. Please keep it up.

[eddiebheara]

I just watched the video, read your explanations in your answers to a few comments, then watched the video again and it makes sense now - you're moving John into bigger control classes as a way to show bigger standard deviations (0 (intervention didn't do anything) -> .3 -> .8 -> 1.3 -> 2 -> 3 (intervention did a LOT when you hold him up to class that didn't get a super cool intervention)).

I have a couple questions - what happens to the 24 other students in John's class?

Also, can you talk about mean vs. mode here?

Is effect size only calculated in pre-post test designs?

[AISHAFLETCHER]

This was very helpful and straight to the point.

[TheTgirl]

Thank you for providing a simple way to understanding effect size.

[kenapakimia]

This explanation is much much much easier! Thankssss

[mauriciompluz]

Simple and clear. May I, please, translate it to Portugue and use it in Brazil in nonproffit public course. If so, I'd include the credits and the link to original version.

[OnlineSchoolofEnglish]

Thank you! Super clear and concise. I truly appreciate it. 🙂