Is most published science WRONG? – the replication crisis

It seems we have a problem. John Ioannidis said: “There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims.”

Over 70% of researchers have tried and failed to reproduce another scientist’s experiments. 52% agreed that there is a replication crisis.

We live in a world right now where a lot of people say our public policy should “follow the science” in some very important areas – with big consequences. So it does matter that the science should be robust. How big is the problem? What causes it? And how can it be – and indeed is it being – addressed?

The Mallen Baker Show is aimed at all people who see themselves as change makers, with commentary on issues and change movements with a particular focus on climate change and environment, social issues, free speech and corporate social responsibility.

References

Why Most Published Research Findings Are False, Ioannidis 2005

Karl Popper (1959). The logic of scientific discovery. New York, NY: Basic Books.

1,500 scientists lift the lid on reproducibility

What Meta-Analyses Reveal about the Replicability of Psychological Research, Stanley et al 2017

What is social priming?

Reconstruction of a Train Wreck: How Priming Research Went off the Rails

Behavioral Priming: It's All in the Mind, but Whose Mind? Doyen et al, 2012

Contradicted and Initially Stronger Effects in Highly Cited Clinical Research

Scientific Audit – A key management tool (10-20% of R&D funds spent on questionable studies)

Rein in the four horsemen of irreproducibility

The association between early career informal mentorship in academic collaborations and junior author performance

Perspectives on Data Reproducibility and Replicability in Paleoclimate and Climate Science

Wired: Science's "Reproducibility Crisis" Is Being Used as Political Ammunition
2018

‘There is a problem’: Australia’s top scientist Alan Finkel pushes to eradicate bad science

Coverage bias in the HadCRUT4 temperature series and its impact on recent temperature trends

Hoaxers Slip Breastaurants and Dog-Park Sex Into Journals

Science eats its own (Wall Street Journal editorial)

The insidious attacks on scientific truth, Richard Dawkins

[MallenBaker]

My rules for comments in this forum.

You can post your opinion freely in the comments to any of my videos. I like to engage with people on arguments and issues. I will, time permitting, happily engage with comments that are polite and broadly on topic for the video in question. You can still post what you want but if you want a response from me, those are the criteria. If you want to preface your biting critique with an observation of what a total idiot I am, knock yourself out, but I won't respond to those comments any more than I would if someone said that stuff in real life.

Out and out obscenities and personal abuse will be removed. Posts that are just links to other videos you want to promote will be removed. People who engage in bullying other commenters will be asked to stop if I think they are making commenting here unpleasant for others. If they refuse to stop then they'll be removed from the channel.

[RalphEllis]

One of the biggest problems is socio-political interference in science.

As the links below demonstrate, the science of atmospheric CO2 has become a political football, and nothing to do with science. The classification of CO2 as a pollutant by the EPA was childish and absurd - it was purely a political move as CO2 concentrations need to be above 10, 000 ppm to be declared as toxic at work. (Your exhaled breath is 40, 000 ppm.)

And what the EPA declined to tell the public, is that CO2 is essential to all life on Earth. If CO2 drops below 190 ppm, all plant life at higher altitudes will die. And below 150 ppm all C3 plant life dies, even at sea-level, and only some C4s will survive. And the vast majority of animal life will die alongside the plants. (Pre-industrial CO2 was 240 ppm.)

To tell the public only half the story is political propaganda, not science.

The EPA declares CO2 a pollutant:

CO2 is NOT a pollutant:

CO2 is the elixir of life:

Ralph.

[davidkelly]

As always, excellent video. The issue for the world public is a gap in perception between what the public think academia do and reality. Most academics, in particular the large institutions, should seek objective truth. In reality, they are seeking grant money and therefore are subject to the age-old human frailties of greed, hubris, arrogance and snake-oil salesmanship. The peer review is also a complete sham as every producer scratch the backs of their mates. Climate change is a good example. At the end of the day it is all about money.

[philerator]

An excellent summary of the issue, very much appreciated.

[nigelmorley]

Great video thanks, Mallen. I’m very concerned about science generally, it’s becoming increasingly political, which undermines its value and trustworthiness. It’s basically trashing its own brand which is very difficult to reverse.

[annatanneberger]

Thanks Mallen. Once again, all your hard work is appreciated. And very well delivered so it's easy and fun to listen to. 
Elaborating on your point, it is also helpful to point out to people that science is at the cutting edge - and knowledge increases by leaps and bounds; not only as one scientist stands on another's shoulders, but also as one scientist disproves another. And often people run amok with wrong conclusions from findings, rather than wrong science. The science most people are familiar with is nutrition. You've heard people complain "one day butter is bad for you and now it is better than margarine. WTF?" It is then difficult to explain that it was just the conclusions and interpretations and thus the recommendations that have changed.

People think all scientists agree. Nothing can be further from the truth. They fight like ferrets in a bag. Or at least that was the case when I was involved decades ago. Even archaeologists would argue viciously about, erm.. when the stone age ended or something. Maybe now they are all cowed by the Woke crowd and step in line? I don't know. Would be sad.

[petersutton]

Nice video Mallen, calm and balanced once again.
I don't think science is broken but I'm pretty sure the political use of it is, if not totally broken, at least highly questionable. The only answer to that problem is a return to integrity and honesty in public and political life and significant penalties for those who stay from the straight and narrow.

[andyh]

Computer code commenting. We're looking at you Neil Ferguson!

[Merlin]

I'm long out of any involvement in academic research, so I can't speak to current practice.
My experience was that people in Experimental Psychology were very happy to supply other researchers with their data and all necessary metadata to facilitate replication. Although the quantities were much smaller then, the effort required to supply it was probably greater than now, when it all sits on your laptop and can be sent to anyone without leaving your chair.
I was surprised to hear a researcher (I think Mr Mann) once say that he would not share his data, because someone wanted it to criticise him. People were sometimes circumspect about it before publication, but once a paper was out I think they were just pleased when other people showed that much interest.

Your petabyte of atmospheric data does pose a greater problem. Climate modelling is a really extreme example of large datasets. But if these models are going to be used to justify the annihilation of human civilisation at a cost that could run to petadollars, surely it's not too much to ask that someone give it the once over. I'd be very surprised if the Americans were the only people capable of running these models. Even if Chinese supercomputers are not as good as the US ones (and I can't see why they wouldn't actually be much better!), unless they are really generations behind, differences show up in the time it takes to run the model, or the number of runs you can do in the allotted time.

Your quote didn't say, they don't get checked, merely that it's a big task. I'd always assumed all these modellers did check each others models. If they don't / haven't, I wonder if it's because no one has yet created a model that's worth the trouble? (Sounds snide, but as far as I can see, models are still very much a work in progress. And the guy saying that a petabyte of data would deter a comparable centre in another country, seems at least as offensive - or maybe disingenuous!)

On an equally cynical note, maybe the (richer) governments don't actually care whether the results are correct. It seems that Mr Johnson certainly sees money to be made from the green technology industry and provided we all go green, no one will ever know whether it was actually necessary - not in our lifetime. I'm not suggesting conspiracy, but there's a hell of a lot of money to be made and greed is ubiquitous.

[yengsabio]

Some years ago, BBC Radio 4's Analysis podcasted "The Replication Crisis." It's a good podcast discussing the matter.

I'm glad this topic made the surface again. Thanks for doing so, Malen!

Lots'a love, cheers, & Mabuhay, from tropical Philippines!

[KevinBalch-dtot]

I would like to see the peer review correspondence included with research articles. This would include the comments made by peer reviewers and how they were addressed by the authors. I used to perform nuclear power plant accident analyses. My calculations were reviewed by an independent reviewer. The calculation included an appendix that listed the comments made by the independent reviewer and how the issue was resolved. This section was extremely valuable in making sure the calculation was accurate but also useful for a new engineer updating the calculation in the future.

[vargon]

Loved the presentation until the end. Dawkins' confidence in his paradigm is no different from the other publishers mentioned at the beginning of the program. There's no duplication of results in the "science" of evolution. The fact that slime molds and starfish have DNA does not demonstrate that all living things are related. Nobody can explain where the information embedded in DNA came from. Likewise, There's no explanation how the mechanism for decoding that information into various molecular machines arose.
Rather than coming up with new information using random mutations as a driving force, it's been shown that the variations in dog breeds, for example, come about by removing or shutting off certain parts of the DNA code.

[jyutzler]

What about more support for "failed" experiments so that these things are still published? If they are still valuable scientifically, they should not be dismissed out of hand.

[daveshongkongchinachannel]

This is something I've long suspected. There are similar examples in society in general and this issue is not restricted to science. In business too, we put great emphasis on data and people are under constant pressure to perform. The larger the organisation the worse the problem and you end up with entire teams of people focusing entirely on the numbers or KPIs or whatever it is, to the neglect of the original reason for putting this targets in place in the first place. This stifles initiative and creativity and erodes the bedrock of a free and open society.

[mabelheinzle]

Again, an informative, interesting and educational content - thank you 🙏 Mallen

[charlesphillips]

IMHO the main problem is that there is no reward and potential risks in trying to reproduce the results of others.

If one manages to replicate the findings, then nobody gives credit for that work. If one fails to replicate the findings then either one has made an enemy or one is considered an idiot.

Important findings should always be replicated by another team and the second team should be rewarded for that work.

[charliewalters]

Great channel Mallen - thankyou. It's only science after it is replicated, for me. So shall we stop calling something scientifically it is replicated.?
Which is - of course - different to actually caring about how we curate the what we leave to those who follow

[martingreen]

I could never get the "right" results at university, so I cheated, and so did my friends. So we are teaching students to cheat on results not find out why the results differ.

[chrispawlus]

Good science is a rare thing these days. There are so many ways that results/conclusions can be compromised. The integrity of the scientist is key.

[stanwizz]

Sorry Mallen but has there been any peer review science on what size is a virus and how many pass through a piece of cloth when inhaling and exhaling.