The BROKEN system at the heart of Academia

Follow me:
Behavioral Science Instagram: @petejudoofficial
Instagram: @petejudo
Twitter: @petejudo
LinkedIn: Peter Judodihardjo

Good tools I actually use:

[ericpenrose]

The peer review that matters is the criticism that happens after a study has been published to the wider community. Not the three sleep deprived researchers who skimmed your paper on their lunch break.

[zray]

I've reviewed 11 papers in the last 2 years, and I don't know if I want to laugh or cry about being "top professors" and "highly paid".

[melimsah]

I'm not in academia but I care about science and education so deeply. One thing I hear about is how important it is that experiments are able to be replicated, but that it's often frowned upon or flat impossible to replicate a study, or if you do and find that it doesn't replicate, it's assumed you're in the wrong rather than the original researchers.

[emockensturm]

A huge part of this that you’re missing (especially in STEM fields) is not that faculty members aren’t compensated for peer reviews, they are also not credited. They get no recognition from the community or those in charge of their promotions. When the personal benefit of doing something is nearly zero, a cost-benefit analysis tells you to skip it. I think a big reason people don’t say no when asked to review a paper is that they are likely familiar with the journal editor asking for the review. Saying no makes you look pretty selfish.

[ferusskywalker]

There's a huge issue with confirmation bias in the papers as well. If the paper directly goes against what someone believes to be true, it's going to be looked at a lot harsher and more rigorously than something that agrees with the reviewers belief.

[doctorlolchicken]

I did peer reviews in the early 90s and it was bs then. Didn’t expect it to improve. In my field in business we are required by the government to have independent verification by a third party, which include replicating a lot of the data analysis, or at least studying the data closely. I think for research involving data some examination of the data should be required.

[Draxis]

For someone who has been in academia and gave up will likely adore this video.
I always grew up wanting to become a scientist or someone who worked at academia. But when I finally made it, I've found there was only Egos vs Egos. An eternal clash of personalities over the most petty reasons possible. I've made the terrible mistake of confusing knowledge with wisdom, and thought that the people who were in academia were "elevated" due to their vast knowledge.
When it was all the same monolithic color of human ignorance there was in society. In sum, I find it hard to solve the BIAS problem without an external factor added in. However, that may be too costly to implement given the complex nature of scientific work.

[warrenkruger]

I have done a LOT of peer review (100s). Everything you say is true, but you missed the largest cause of bias. Because your own work is is judged by citations, papers that cite your work a lot tend to be favored. The competitor issue is generally less of a problem with papers as most journals allow you to exclude reviewers. I do think better peer review would occur if Reviewers were compensated.

[gretalaube]

Did research for 30 plus years. This is why when everybody cheers: "Because SCIENCE!" I laugh. The blind leading the blind. Oh, and don't forget egos!

[lisleigfried]

Making stock footage of yourself to save time editing is genius

[lindasegerious]

I have published research, and I was offered to be a peer reviewer for an academic journal, but on a topic where my background and level of expertise are minimal. I mentioned that. Still they insisted. In the end I never accepted, but the fact that they insisted opened my eyes about the reliability of the peer-review system.

[RoughPerspective]

As a nobody in psych, meaning that I have yet to conclude my MS, I was offered a peer reviewer "job" by a high impact journal. Of course I never took up the opportunity because... it would be unethical to say the least...but it speaks volumes about how stringent the process is. Just wanted to put it out there, love your videos ❤

[UnsafeKibble]

Peer reviewed is further problematic because this third party system is tied to advancement. Some university tenure systems will not accept a publication in an open access journal as good enough for tenure and promotion considerations. Instead, they demand publication is a prestigious peer review journal. Some journals charge thousands to make the articles that they publish available as open access.

[theysisossenthime]

I'm not involved in a scientific peer review process, but I am directly involved in engineering peer review. I see many similar issues between the two. The biggest issue I continuously struggle with is that a significant population of people that I've worked with don't see the value added by the peer review process. Even after seeing how engineering projects have literally been saved by peer review, too many just want to wave a magic wand over it and move on.

[doctorgumby]

The problems with peer review you mentioned are well known and have been for decades. I would be interested in alternate solutions. In my opinion, peer review is the best option we have. That being said, one improvement I hope is implemented soon is that verification studies are promoted alongside novel studies. You can't have a complete scientific process without rigorous verification of the results.

[cipaisone]

Before leaving academia, I reviewed dozens journal in my niche area of material science. I do agree with all you said, and I want to add another point, that concerns material science and perhaps not so much other areas.
In material science, many works often involve multiple area of competence, as a material or phenomenon in material science might be characterised with a number of methods that are very different, and theories involved to explain different characteristics can require very different background too. Often, No reviewer can possibly have a extensive-enough background to verify all the various data presented, let alone the theoretical speculation proposed by the authors. Even with 2-3 reviewers, it is often the case that none of them is able to judge some of the presented data. As such, it is very likely to have works where very large portion of the study was simply neglected by the reviewer, due to lack of knowledge.
Although this is not generally concerning major results of the study, but rather supporting data, this situation leads to a lot of garbage (unreliable) data produced on scientific journal in the field of material science.
Over the time, this poison the entire literature, making it very difficult to find reliable scientific data for even basic parameters. In fact, data on simple, “reference”, materials has become so corrupted that it is virtually impossible to find reliable sources on most of material science journal.
I believe that this trend will leave to the collapse of the entire field of research of material science, over some decades, as it will become clear that most of the scientific data published are unreliable….and perhaps that would be a good change, as it will force material science to focus on reliability rather than fanciness .

[phillustrator]

Edit: I wrote this comment before watching the video thinking Pete would not dare go where I went. Props to you Pete for your courage.
Peer review for some (a lot) of researchers is just a way to either make you cite their papers, or make sure you don't publish in a "prestigious" journal. If you think about it, them being not paid for their work means they have no other incentive, Your peers are also your competitors. We need to make peer review open, not anonymous, and voluntary. Also: reviewers need to disclose ALL their conflicts of interest, not just the financial ones. Working on the same topic and being upset that you were scooped is a conflict of interest too. We also need to abolish fancy journals like Nature because all they do is incentivize fraud. I have much more trust in a paper published in a mid-tier journal than a Nature paper, because I have seen how the Nature sausage is made.

[DarianCabot]

As a layman, not in academia, I find these videos eye-opening. I have always naively assumed the peer review process worked. I'm now questioning all those statistics quoted by articles and experts about "research has proven XYX". I'm interested in hearing more about peer review alternatives. Thanks for your videos!

[Tcheera]

Yeah... I worked at one of the top centers of excellence in the nation for my field and on a project that had really unique datasets. Usually just by the abstract, the intro, or the participants / methods section ANYONE in the field would know -- maybe not who the first author was, but they would know whose project it was who was literally the top author in the field, and just about anyone affiliated would get accepted.

I was first author for two submissions during my PhD early on before I was even a dissertator on two publications -- it was funny because I was both forced to "dumb down" both publications and not use the best stats methods for my multi-time point studies so that people in the field would be more likely to understand it (i.e. I was going to use HLM but was encouraged to just use Multiple Regression because they knew that none of the reviewers would probably be trained in HLM). And it was also clear that everyone would know whose study it was -- the field was run and reviewed by such a small cluster. You look at the dataset across studies -- there's only one dataset that could match the one that I used LITERALLY in the whole world based on the description. And I think that's kind of unfortunate in some ways because I think my study was fine -- but I remember my professors being -- I don't know maybe feigning being shocked that both of my studies got through on the first round. I remember kinda not being shocked because I thought -- well who is going to reject these studies by them?

*shrug* -- even if they were all double-blind, they would be completely known. And even with the feedback from the reviewer a lot of times we would know who the reviewers were LOL. I mean, the reviewer challenges you on contradicting studies from the past "what about the studies that say x, y, z!" And they're so passionate -- and we've kind of already addressed them, but it's clear that person is the reviewer because they're upset we've kind of found findings that are contradicting so we have to put in a line giving them props and explaining how their work was INTEGRAL to our current theory even though it was about children and not adolescents.

[mortifinkenbein]

One problem with peer review is the sheer number of papers that have to be reviewed. It became very obvious during the Covid-19 pandemic, when every small medical insitution pushed out two papers a month because they saw a chance to make a name for themselves.