The Problem with Ranking Universities (League Tables) - Sixty Symbols

We discuss the problem with ranking universities using so-called "league tables". More links and info below ↓ ↓ ↓

In this video we hear from Professor Philip Moriarty, Meghan Gray and Mike Merrifield... All are from the University of Nottingham and are expressing their personal views on league tables.

This video features scientists from The University of Nottingham School of Physics and Astronomy

Also in this little collection of interviews...

In the interests of fairness, here are some popular league table websites...

Sixty Symbols videos by Brady Haran

[Tmpp]

The "nobody smiled"" argument is SO important and I have a feeling not a single league table or equivalent ranking system takes it into account. If you're gonna be spending several years in a learning institution, it doesn't matter how prestigious or well-equipped it is if you're having a miserable time with miserable people all the way through! Obviously there's personal opinion in this, but I'd take a #12 university where most people are enjoying the daily grind and the atmosphere is welcoming and inclusive over a #1-3 university where every day is tense and stressful a million times out of a million. 😸

[everythingbrassorange]

Congrats to Professor Merrifield on retirement!

[doctorscoot]

Re: the sports table analogy. It’s like the premier league wasn’t just decided on win/draw/loss points and goal difference, but it also included, all equally ranked with the points: a style of play rating; a jersey design rating; a sponsorship dollar index; best signing of the season rating; a fan song contest; and a stadium comfort ranking.

[yoram_snir]

Immediately sent it to my teenage son. He is so fixated on those tables. I keep trying to tell him that there is no way he can correctly decide what’s good for him based on some website rankings.

[nhmw]

In a race, every news source will report the same results: who won, the final score, elapsed time, etc. Results are objective, at least within the framework of the sport's rules. League tables are obviously subjective when the tables from the various sources don't agree with one another in the least.

Additionally, in sports, trends tend to be the norm. The are plenty of exceptions, of course, but good teams or good athletes tend to stay good over a period of time. No athlete who has never been able to run 100 meters in less than 12 seconds is going to suddenly one day beat Usain Bolt in a race. Similarly, how can a University or an academic program be ranked number four one year, twenty nine the next, fifteen after that, and so on? It's not reasonable. Nor objective.

Lastly, how can two institutions or academic programs be ranked, when one of them is research-oriented while the other is for future educators, for example?

[desicmanifold]

This is really potent the way it's edited. Can't wait for the discourse when it's released in full!

[mikechiu]

The fact that there are so many league table publishers and their rankings differ so wildly tells you how subjective (and pointless) these numbers are.

[wvseahawks]

It’s hard because there’s more than one objective. If I had a university where it’s poor at job placement for undergrads but the grad students find a cure for a different cancer each year and another university has bad research but 100% job placement, which is better? Giving an equal score to both isn’t helpful

[ScientiaHistoria]

As Prof. Merrifield alludes to at 11:30 and 17:40, the fundamental quantitative flaw is they collapse multiple value dimensions into a single metric — that’s not just difficult, it’s inherently impossible without an objective conversion factor among the input variables. “We looked at new cars’ acceleration and petrol-efficiency and ranked the best cars as follows…” not doable unless you know the *subjective* value of a unit of acceleration per liter/kilometer of fuel. And that’s subjective to the consumer, just as students are subjective consumers who value different inputs to the rankings differently. If you had interviewed the Nottingham economics department they’d probably have ranted about this flaw more than about error bars.

[Harlequin_]

I appreciate that you always try to get a bit of a rise out of professor Moriarty, and that he always gives you one. The man is passionate!

[phrankster]

I've been ranting about the nonsense of league tables since I first applied to university. You only have to read a few before you notice the lack of logic to them. What's far more important to students and wider society is reputation. If your uni has a good rep then people know about it, particularly with certain subjects. If you say you studied medicine at Nottingham or classics at Oxford, or art at The Slade School, or biology at Imperial; people understand that's impressive. Observing talented students choosing universities, for them it's about the city or campus, how close they are to home, and just the vibes from their visits. So I wouldn't worry about it. I'd worry more about getting the vibes right.

[robertbrown]

Brady does a top notch job playing devil's advocate. Great way to handle interviewing about it. That being said, as someone who works in higher ed, university rankings publications are BS and people internally all know it. We play the game for public perception and nothing else.

[macalmy]

One big flaw with Brady's Olympics analogy, even with more subjective events like gymnastics, is that very few people, if any, make important life decisions based on who won the gold medal. Damn straight I'd want error bars on the gymnastics scores if people were using those results to decide something like what university to attend.

[erikoui]

Adding to this, universities sometimes optimize for the metrics of the league tables and ignore aspects that are important but not considered.

[GoatzAreEpic]

I just wanted to say I really appreciate Brady's critical points around 8:50. Really the best interviewer I have ever seen.

[henrikskott]

Excellent video! Especially the bit about why the numbers change, because how the ranking is caluclated changes from year to year.

[edgeeffect]

I've recently been grumbling quite vigorously about job interviews and recruitment processes recently. And many of my grumbles have incredible resonance with the many of the issues raised here. Phil's comments about peer-review and grant proposals and actually engaging with the material really struck a cord.
Also when you're asking Mike if he could come up with an alternative league table is similar to a Veritasium video a while back, where Derek was talking about expertise and how certain environments just aren't "valid enough" to ever allow a real expert to exist.

[bipolarminddroppings]

Always thought that the fact that at primary and secondary school level the worse a school performs the less resources they get is ass backwards.

Surely you should want the best teachers at the worst schools? And that costs money.

Never realised they were such a problem at higher education level.

[srwapo]

There's two things I dislike about these kinds of lists:
The schools with the highest rating would get the "best" students, so I would assume they would scale higher for things like average grades.
You're looking at the schools rated at the highest when there are like 4000 higher ed schools (in America). So people are comparing the, like, 0.25%of the schools people actually go to.

[ryanf]

Great video with great challenges from Brady. While I completely recognise the flaws in these rankings, I take issue with 'don't base your decision on the league tables at all'. Rightly or wrongly, the quality of your university matters to many employers (particularly for your 1st job out of university). You could ignore the league tables entirely and study at a low ranked university on issues that matter to you but some employers will bin your application as a result.