Your Favorite Research Is (Probably) Wrong: The Replicability Crisis

Ever heard of the marshmallow experiment? The 10,000 hour rule? How about the Dunning-Kruger effect, the Stanford prison experiment, or willpower fatigue?

These are some of the most well-known pieces of research from the social sciences. And they all share one problem: they're wrong. Or, at least, they're really misunderstood. It's shockingly common for all kinds of holes to be found in research that becomes popular, but popular belief in the theory is rarely updated.

In this video we're exploring social science's Replicability Crisis, and going on a tour of famous research that was later invalidated.

Chapters:

00:00 Intro
01:16 Dunning-Kruger
02:52 Backfire Effect
05:20 Marshmallow Test
07:18 Stanford Prison Experiment
09:04 Willpower Fatigue
09:40 Critical Positivity Ratio
10:37 10,000 Hour Rule
12:27 The Replicability Crisis

Cited Research:

Kruger, J.; Dunning, D. (1999) "Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments.”
Nuhfer, Edward; Cogan, Christopher; Fleischer, Steven; Gaze, Eric; Wirth, Karl. (2016) "Random Number Simulations Reveal How Random Noise Affects the Measurements and Graphical Portrayals of Self-Assessed Competency.”
Shoda, Y., Mischel, W., & Peake, P. K. (1990). Predicting adolescent cognitive and self-regulatory competencies from preschool delay of gratification: Identifying diagnostic conditions.
Tyler W. Watts, Greg J. Duncan, Haonan Quan. (2018) Revisiting the Marshmallow Test: A Conceptual Replication Investigating Links Between Early Delay of Gratification and Later Outcomes.
B. Nyhan , J. Reifler. (2010) “When Corrections Fail: The Persistence of Political Misperceptions.” Wood, T., Porter, E. (2018) “The Elusive Backfire Effect: Mass Attitudes' Steadfast Factual Adherence.”
Brown NJ, Sokal AD, Friedman HL. (2013). The complex dynamics of wishful thinking: the critical positivity ratio.
Haney, C., Banks, C., & Zimbardo, P. (1973). Study of Prisoners and Guards in a Simulated Prison.
Baumeister, R.F. (2002) Ego Depletion and Self-Control Failure: An Energy Model of the Self's Executive Function.
Carter E.C., Kofler L.M., Forster D.E., McCullough M.E. (2015) A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource.
Brown N.J., Sokal A.D., Friedman H.L. (2013). The complex dynamics of wishful thinking: the critical positivity ratio. Ericsson, A. K. (2008) Deliberate Practice and Acquisition of Expert Performance: A General Overview.

You can follow me here:

[wendyperrotti]

Love this... It’s such an important topic. The trouble with a digital world is that incorrect information doesn’t go away. People want to feel better about themselves and about their world but that doesn’t mean they want to become researchers or journalists themselves. When the top 5 hits on a google search tell them something, why would they look further?

[TallulahGuard]

It's crazy to see you rather than just hear you on the podcast!!! :))

[adrianthomas]

Very interesting - I’ve heard the same thing said about patents - a patent is published and in many cases the discovery cannot be replicated. We need to be cautious and thoughtful.

[NondescriptMammal]

Even if the 10, 000 rule had been valid for violinists, I am baffled why any scientific-thinking person would just assume that this would magically transfer to all other activities.

[patricialong]

Fascinating and thought-provoking information. Thank you for sharing!

[chandanawatagodakumbura]

Thank you, Forrest, for that important discussion.

[erikafranziska]

As usual, this is well-researched, eloquently presented and beautifully chosen for its relevance and topicality. Thank you very much for putting this together. As a psychologist-in-training, I do wish you had reiterated at the end how many important, correct and interesting studies there are regardless of the very real and troubling replicability crisis.

[ParentSmileandGrow]

I loved this! Such a needed topic, please share more! 😍

[amazinggrapes]

I also heard you about the Stanford prison experiment but what I heard was that the participants were volunteers. What kind of person jumps at the chance to be a prison guard? Well, exactly who you'd think
I think it would be worth testing further the idea that power attracts the corrupt as opposed to corrupting those in power

[amazinggrapes]

I'd heard of the correction of the marshmallow test before. Turns out, children are more likely to wait for a greater reward if their experiences have led them to believe they're likely to actually get that reward. Whoda thunk

[chrisschneider]

Crazy. Dunning–Kruger is something I have used to explain why some people are so confident of their opinion but completely lack the knowledge to be able to understand how incorrect they are. It seems to be a common issue at the moment, is there another theory that explains this better?

[amazinggrapes]

The ego depletion thing... Anything that takes energy becomes harder to do when you have less energy to draw from. Not surprising

[jessekhaophilia]

Late to the party, but I would say this: taking into account socio-economic status does not necessarily include the marshmallow experiment. It could simply mean that these traits persist through generations, i.e. socio-economic status is lower because their parents, grandparents, etc. also had time preference issues.

[fred]

I have learned a great deal from this, thank-you

[kyle]

Will power depletion could be glutamate up regulation which does effect decision making

[kerimalsandi]

are there any famous psychological studies that are known to be highly replicable? or do they not exist?

[ericsuarez]

I have seen the Dunning Krueger effect on theologists explaining why science is wrong about god so I know is anecdotal but it really does happen, maybe not as often as we with our short perception may think

[poetmaggie]

I think Human Nature is as complicated as the Climate and like the Climate it is not possible to boil it down into a simple jell.

[poetmaggie]

The Social Science and psychology fail to include the entire nature of humans.

[BantuNiLo]

I never trust research I trust reality