MINI LESSON 7: P-Values and P-Value Hacking: a simplified lecture.

We saw that 1) many metrics are stochastic, 2) what is stochastic can be hacked. This is the simplification of my work showing that "p-values are not p-values", i.e. highly sample dependent, with a skewed distribution. For instance for a "true" P value of .11, 53% of observations will show less than .05. This allows for hacking: in a few trials a researcher can get a fake p-value of .01.
Paper is here and in Chapter 19 of SCOFT (Statistical Conseq of Fat Tails):

[leswhynin]

This might be my favourite probability mooc because it starts with "if you don't know p-value means, forget about it"

[mac]

Was not aware of electronic version provided free. Well done. I feel better now about listening to Fooled By Randomness on YouTube. Since then, I have purchased all 5 of Incerto. Listen daily. Thanks for teaching!

[yunastic]

In medicine, repeat the same clinical trial a sufficient number of times, then only publish the positive studies (p<0.05) that will happen 5% of the time and meta-analyse the positive studies. It’s magic! I love more and more industrial medicine. 😂

[praneshsaisridhar]

Maestro, your series of videos have changed the way I approach learning basic statistics. 🙏

[jonathanelkefif]

The content we do not deserve... thank you for helping many of us stand on the shoulders of giants. I'm certain you bring much relief to those few who scar themselves for having their skin in the game.

[castlehedge]

Another perfectly executed debunk of an overly used statistical tool. Very well done.

[pioneercolonel]

For anyone interested to see a simulation, the following setting in R gives you the P-hacking dilemma when the real p-value is 0.11.

bias <- 1
n=10
p_value_vec <- sapply(1:1000, FUN = function(x) t.test(rnorm(n = n), rnorm(n = n, mean = bias))$p.value)
print(mean(p_value_vec))
hist(p_value_vec, breaks = 20)

What this simulation shows is basically if everything is done correctly, on a single experiment there is an above 50% chance that an observed p-value of 0.05 over 10 samples in each group has a true p-value of 0.11

As long as the real p-value is the same, this observation is independent of the sample size.

For example for a n=100 setting

bias <- 0.3
n=100
p_value_vec <- sapply(1:1000, FUN = function(x) t.test(rnorm(n = n), rnorm(n = n, mean = bias))$p.value)
print(mean(p_value_vec))
hist(p_value_vec, breaks = 20)
print(sum(p_value_vec< 0.05))


Thank you, Nassim for bringing this to attention.

[mahadevaniyer]

Hi Taleb! Awesome exposition! 😁👍 My intuitive take away is any sort of inference, even from a large data set, if expressed as a parameter must necessarily be itself stochastic and needs to be interpreted with caution!!! 😁👍

[travisrobson]

I'm loving these mini lectures, such powerful statements made from the basics. I started them not too long ago and the piqued my interested to grab a couple of your books. Making my way through my first one Fooled by Randomness. It's been a while since I've enjoyed a book this much.

[ractheworld]

The great teacher of our time. Thank you!

[anrm]

Been waiting for this. Always love it when Prof Taleb uploads

[KarimTraderFutures]

Merci beaucoup pour les explications Monsieur Taleb, et pour le lien.

[coAdjointTom]

Sir, it is not just psychologists who abuse this notion. A former employer (with a PhD in statistics) would choose when to use bonferoni Holm adjustment based on the likelihood our client would stop our funding based on the results of our spurious regressions. Data scientists in the wild don't like to think about randomness

[rikeyjo]

Thankful as always for calling out BS!

[thomaskaminski]

The vast majority of biosciences like them too, it's a huge problem. Thanks for the lectures and the books.

[wldymir]

Friend ! I finished fooled by randomness last night, good writing style and valuable lessons, I will read the other ones. I'm playing around with a small equity options portfolio for hedging and speculation purposes, long volatility, for fun (and hopefully profits), it's really nice to get your insight.

[andreischiopu]

il miglior professore del mondo. Grazie Signore

[berberidian]

Ευχαριστούμε Δάσκαλε!

Every lecture you put out makes me more an more interested in probability/statistics.
(An appreciation I unfortunately did not develop as much when in school / university)

[colour_thief]

Anyone looking it up in the book, it is actually page 349.

[backtoemocovers]

"Probability... is the acceptance of the lack of certainty in our knowledge and the development of methods for dealing with our ignorance. Outside of textbooks and casinos, probability almost never presents itself as a mathematical problem" (Taleb, 2004, p. x). That is one of my favorite part of his book, Fooled by Randomness.