Has Generative AI Already Peaked? - Computerphile

A new paper suggests diminishing returns from larger and larger generative AI models. Dr Mike Pound discusses.

This video was filmed and edited by Sean Riley.

[miroslavhoudek]

As a sort of large trained model myself, running on a efficient biological computer, I can attest to the fact that I've been very expensive over the decades and I certainly plateaued quite some time ago. That is all.

[tommihommi]

generative AI has destroyed internet search results forever

[Rolox]

So refreshing to hear grounded academics talk about these sorts of things and take realistic look at what’s happening. Feels like everyone wants to say anything about generative AI

[marcusmoonstein]

You've just described the problem being experienced by Tesla with their self-driving software. They call it "tail-end events", which are very uncommon but critical driving events that are under-represented in their training data because they're so rare.

Tesla has millions of hours of driving data from their cars, so the software is better than humans in situations that are well-represented in the data such as normal freeway driving. But because the software doesn't actually understand what it's doing, any event that is very uncommon (such as an overturned truck blocking a lane) can lead to the software catastrophically misreading the situation and killing people.

[Posiman]

This is the computational side of the argument for AI peak.

The practical side is that the amound of existing high-quality data in the world is limited. The AI companies are already running out.

They theorize about using synthetic data, i.e. using model-generated data to train the model. But this leads to a model collapse or "Habsburg AI" where the output quality starts quickly deteriorating.

[michaelujkim]

even if you took the whole internet as a dataset, the real world is orders of magnitude more complicated.

[LesterBrunt]

People just completely underestimate how complex human cognition is.

[leckstr]

10:37 "starts to hallucinate"

I recently heard it expressed that AI doesn't "sometimes" hallucinate. AI is always hallucinating and most of the time its hallucination matches reality/expectation.

[RobShocks]

Your ability to articulate complex topics so simply with very little cuts and editing adlib is amazing. What a skill.

[ArtePluviali]

"Man is not able to create even a worm, but creates gods by the dozen"
"Having failed to achieve what they wanted, they pretended that they wanted what they had achieved"
Michel de Montaigne

[allgoodhandlesweretaken]

also a lot of people seem to think that OpenAI came up with some novel way of engineering this stuff when in reality most of the progress we have seen is just the result of more compute and an increase in parameter count and dataset size. seems unlikely there will be an exponential curve when increasing these two is so hard and expensive.

[tommydowning]

I love this content where we get to delve into white papers with the convenience of a youtube video, not to mention with the genuine enthusiasm Mike always brings to the table.
Great stuff, thanks!

[djdedan]

I’m not a child development. Specialist so take this with a grain of salt but What’s interesting is that you can show a child one image of a cat. Doesn’t even have to be realistic and they’ll be able to identify most cats from then on. What’s interesting is that they may mistake a dog for a cat and they will have to be corrected but from then on they will be able to discern the two with pretty high accuracy. No billions of images needed.

[TheGbelcher]

“If you show it enough cats and dogs eventually the elephant will be implied.”

Damn, that was a good analogy. I’m going to use that the next time someone says that AI will take over the Earth as soon as it can solve word problems.

[peterisawesomeplease]

I think a key issue is we are actually running out of high quality data. LLMs are already ingesting basically all high quality public data. They used to get big performance jumps by just using more data. But that isn't really an option anymore. They need to do better with existing data.

[ekki]

As a bioinformatician, I will always assume that the exponential growth will plateau sooner rather than later. Sure, new architectures may cause some expected exponential growth for a while, but they will find their ceiling quite fast.

[supersnail]

Im surprised 'degeneracy' wasnt also mentioned in this - basically that as more AI generated content leaks into the dataset, further training could actually lead to worse results. There are ways of encoding the output to evidence that the data was generated, but that likely wont hold up if edits were made to the data prior to it entering the training corpus.

[marcbecker]

This is totally off-topic, but the quality of the X and Y axes of the graph at 5:43 is just stunning.

[PristinePerceptions]

The data we have is actually incredibly limited. We only use mostly 2D image data. But in the real world, a cat is an animal. We perceive it in a 3D space with all of our senses, observe its behavior over time, compare it all to other animals, and influence its behavior over time. All of that, and more, makes a cat a cat. No AI has such kind of data.

[ohno]

Everyone remembers the people who wrongly said the internet wouldn't transform our lives, no one remembers the people who correctly said the segway wouldn't transform our lives