269 - Good vs. bad science: how to read and understand scientific studies

This special episode is a rebroadcast of AMA #30, now made available to everyone, in which Peter and Bob Kaplan dive deep into all things related to studying studies to help one sift through the noise to find the signal. They define various types of studies, how a study progresses from idea to execution, and how to identify study strengths and limitations. They explain how clinical trials work, as well as biases and common pitfalls to watch out for. They dig into key factors that contribute to the rigor (or lack thereof) of an experiment, and they discuss how to measure effect size, differentiate relative risk from absolute risk, and what it really means when a study is statistically significant. Finally, Peter lays out his personal process when reading through scientific papers.

We discuss:
0:00:00 - Intro
0:00:18 - The ever-changing landscape of scientific literature
0:03:19 - The process for a study to progress from idea to design to execution
0:06:39 - Various types of studies and how they differ
0:19:14 - The different phases of clinical trials
0:28:32 - Observational studies and the potential for bias
0:49:27 - Experimental studies: randomization, blinding, and other factors that make or break a study
1:03:49 - Power, p-values, and statistical significance
1:16:21 - Measuring effect size: relative risk vs. absolute risk, hazard ratios, and “number needed to treat”
1:26:39 - How to interpret confidence intervals
1:34:15 - Why a study might be stopped before its completion
1:43:01 - Why only a fraction of studies are ever published and how to combat publication bias
1:53:45 - Why certain journals are more respected than others
1:57:54 - Peter’s process when reading a scientific paper

--------
About:

The Peter Attia Drive is a deep-dive podcast focusing on maximizing longevity, and all that goes into that from physical to cognitive to emotional health. With over 70 million episodes downloaded, it features topics including exercise, nutritional biochemistry, cardiovascular disease, Alzheimer’s disease, cancer, mental health, and much more.

Peter Attia is the founder of Early Medical, a medical practice that applies the principles of Medicine 3.0 to patients with the goal of lengthening their lifespan and simultaneously improving their healthspan.

Connect with Peter on:

Subscribe to The Drive:

[PeterAttiaMD]

In this episode, we discuss:
0:00:18 - The ever-changing landscape of scientific literature
0:03:19 - The process for a study to progress from idea to design to execution
0:06:39 - Various types of studies and how they differ
0:19:14 - The different phases of clinical trials
0:28:32 - Observational studies and the potential for bias
0:49:27 - Experimental studies: randomization, blinding, and other factors that make or break a study
1:03:49 - Power, p-values, and statistical significance
1:16:21 - Measuring effect size: relative risk vs. absolute risk, hazard ratios, and “number needed to treat”
1:26:39 - How to interpret confidence intervals
1:34:15 - Why a study might be stopped before its completion
1:43:01 - Why only a fraction of studies are ever published and how to combat publication bias
1:53:45 - Why certain journals are more respected than others
1:57:54 - Peter’s process when reading a scientific paper

[albrackets]

When you read about diet and nutrition studies in the NYTimes or Guardian, often the “experts” they talk to speak in the lexicon of lobbyists

[sebacatana]

Oh this is just amazing! A very much needed podcast in today's information storm. Thank you guys!

[zuzannagradek]

I'm currently in my second year of Dietetics and not ONCE has any lecturer mentioned how to read and interpret literature. This video has been very useful, thank you :)

[kimdecker]

What a valuable pod. I confess, a lot went over my head. But I believe I'll be a better consumer of science for having listened. I hope a lot of people do the same! Thank you!

[loriwilliams]

This summarizes my Masters degree in Health Research Methodology beautifully. Brilliant. Although I disagree with your use of "open label" - to me this means unblinded, not non-randomized.

[lilytea]

0:01: 📚 The process of a study involves forming a hypothesis, designing and executing the experiment, obtaining IRB approval, determining primary and secondary outcomes, and pre-registering the study.
9:43: 💡 The passage discusses different types of studies and their hierarchy in medical research.
19:23: 🔬 Observational studies are important but have limitations and should be interpreted cautiously.
28:43: 🔎 The speaker discusses selection bias and recall bias in observational studies, particularly in relation to nutritional epidemiology.
37:23: 🔬 The speaker discusses the limitations and biases in epidemiology studies and the importance of differentiating primary and secondary outcomes.
46:40: :mag_right: Research should be hypothesis-seeking or hypothesis-testing, but it can also be hypothesis-generating. Statistical tools can be used to analyze data in multiple ways, but caution must be taken to avoid false positives.
55:03: :mag_right: Understanding the key factors to consider when evaluating a clinical trial study.
1:04:24: 💡 The concept of statistical significance and power in experimental studies.
1:14:16: 💡 Understanding power, effect size, and risk measurement in research studies.
1:24:43: 📊 Understanding confidence intervals and relative risk reduction is important when interpreting scientific studies.
1:33:37: 🔬 Studies can be stopped midway for safety, benefit, or futility reasons, and it's important to consider the source of science information.
1:43:11: 📚 The process of publishing a study involves submitting it to a journal, peer review, and potential revisions before acceptance or rejection.
1:54:02: 📚 The impact factor of a journal is a measure of its influence based on the number of citations its articles receive, and reputable journals have higher impact factors.
2:00:40: 📝 The speaker discusses their approach to writing scientific papers, starting with figures and legends before writing the rest of the paper.
Recap by Tammy AI

[CrossCultural-cf]

“Understand what it’s like to not understand.”
Richard Saul Wurman,
Co-founder of the TED Conference.

The disease of familiarity afflicts the professor who knew his subject well but couldn’t teach the students because the professor forgot how it was like to not understand.

With utmost respect, I am not sure the podcast creators have a clear idea of what the fund of knowledge of the intended audience of the podcast is. A suggestion I have would be to mention the assumption of what baseline understanding the intended audience of the podcast segment needs to have and summarize the key points to be covered in the podcast episode.

During this podcast, I caught a scattering of concepts, but not a really clear idea of what you thought were important in how to read a study paper.

[charlestoddsullivanforpres]

I'm going to be a downer here - if you don't know the methods being used, if you don't understand the question and relevant background info, the causal pathways, if you don't map out the causal connections, if you don't understand the data generating mechanism, if you don't understand the statistical analysis being done, if you don't map out the relations between variables, then you won't know if you're dealing with a good manuscript or not. If you understand those things but they are not suitably transparent in the paper, then you won't know if it's a good study or not. I'm guessing we're going to get the typical "RCT is the gold standard and Observational studies have bias" line - well, what do you get after randomization is done in a RCT? You get an observational study. Unfortunately, this topic is too complex and you have to learn all of these things if you're going to separate the good from the bad. There just aren't any short cuts, which is too bad because it's a big barrier. Also, peer review is broken.

[TomiRantanen]

Thank you for this topic! And I love that you slipped Feynman in the background.

[careyjamesmajeski]

Science has far greater limitations than outlined here. Gotta read Kuhn, lakatos, feyerabend, laudan, etc. To really understand the nature and scope of empirical inquiry, one needs to commit a few years of study to the philosophy of science, statistical analysis, probability theory, game theory, and probably a few other areas particular to the area you want to assess. There are no shortcuts. It’s work, not a YouTube video. Appreciate the attempt, however. Cheers

[marynone]

Thanks so much!! Been out of college too long. Who developed the parameters for that chart on the number of patients needed in a trial? Is the "significance" level there the same as "confidence" level? Been wishing for a crash course like this!!

[monroetinker]

Thank you for this, it would've been great to have it during graduate school.

[ejtonefan]

I would love to see Peter using the scientific approach for "Climate Change".

[innuendo]

What is needed is AI - no ego, no bias. Sure, that can be tricked, too, however creating strict rules for research and publishing results would help. Templates vary from lab to lab. Deeper standardisation we need ... + emotional and Ego IQ tests on the research teams lol.

[utes]

Unless we're discussing abs understanding the process that got Leqembi, lecanamab, almost uselessly approved for Alzheimers, we're not fully understanding this entire process

[unknownKnownunknowns]

Fascinating topic, but I don't have the time to hear it all. Does anyone have TLDW summary?

[gregredding]

I wish I could get CE credit from your Podcasts!

[lukerestlessstudios]

Hasn’t it also been found that most studies in most fields are garbage?

[danielmccarthyy]

Bad science like the falsified lipid heart hypothesis and the fake statin research?