Ilya Sutskever: 'Sequence to sequence learning with neural networks: what a decade'

Ilya Sutskever full talk "Sequence to sequence learning with neural networks: what a decade" at NeurIPS 2024 in Vancouver, Canada.

"Pre-training as we know it will end" and what comes next is superintelligence: agentic, reasons, understands and is self aware.

NeurIPS 2024 — 2024 Conference on Neural Information Processing Systems.

[RaahilShah]

Ilya’s long pauses prove the scaling hypothesis for test time compute

[TheJayLenoFly]

now this is a christmas gift, Ilya's latest intuition on frontier AI

[GabrielVeda]

Why on earth wasn’t this talk given more time? Good grief.

[marko-o-h]

Ilya needs to talk more and the world needs to listen

[thatthotho]

Love how even his ppt is pure content. Truely an obsessed gifted one

[labsanta]

00:07 - Reflecting on a decade of advancements in neural network learning.
02:52 - Neural networks can mimic human cognitive functions for tasks like translation.
05:05 - Early parallelization techniques led to significant advancements in neural network training.
07:45 - Pre-training in AI is reaching its limits due to finite data availability.
10:27 - Examining brain and body size relationships in evolution.
12:55 - Evolution from basic AI to potential superintelligent systems.
15:14 - Future AI will possess unpredictable capabilities and self-awareness, transforming their functionalities.
17:46 - Biologically inspired AI has limited biological inspiration but holds potential for future insights.
19:43 - Exploring the implications of AI and rights for future intelligent beings.
22:06 - Out-of-distribution generalization in LLMs is complex and not easily defined.
24:22 - Ilya Sutskever concludes with gratitude and audience engagement.

[MultiMojo]

Glad to see Ilya back! Hope he's recovered fully from the OpenAI drama and can go back to doing what he's best at - research.

[edansw]

"If you can't explain it simply, you don't understand it well enough" Ilya is the only one to explain the entire AI domain past-present-and-future with simplictly

[TheRohit]

We need more of Ilya! He is an inspiration for all of us doing AI research.

[ilariacorda]

He is a fantastic speaker, we need to hear more from him. Still his talks should be shared more across

[nootherchance]

Just when the world needed him most he came back!

[kheteshbakoliya]

Man of super short but extremely profound lines. Legend. 🔥

[LightDante]

It's truly impressive how extrodinary Ilya's speeching skills are.

[philtrem]

👏👏👏 The part that keeps resonating in my mind is that they'll be self-aware. And this makes me want to figure out how they could.

[ktxed]

Ilya is the hero we need and the hero we want

[picpic-kc]

Thank you for posting this wonderful talk

[nowithinkyouknowyourewrong]

Here are Ilya Sutskever's main points and conclusions in brief:

## Main Points:
1. **Original Success Formula (2014)**
- Large neural network
- Large dataset
- Autoregressive model
- This simple combination proved surprisingly effective

2. **Evolution of Pre-training**
- Led to breakthrough models like GPT-2, GPT-3
- Drove major AI progress over the decade
- However, pre-training era will eventually end due to data limitations

3. **Data Limitation Crisis**
- We only have "one internet" worth of data
- Data is becoming AI's "fossil fuel"
- This forces the field to find new approaches

## Key Conclusions:
1. **Future Directions**
- Need to move beyond pure pre-training
- Potential solutions include:
- Agent-based approaches
- Synthetic data
- Better inference-time compute

2. **Path to Superintelligence**
- Current systems will evolve to be:
- Truly agentic (versus current limited agency)
- Capable of real reasoning
- More unpredictable
- Self-aware
- This transition will create fundamentally different AI systems from what we have today

3. **Historical Perspective**
- The field has made incredible progress in 10 years
- Many original insights were correct, but some approaches (like pipelining) proved suboptimal
- We're still in early stages of what's possible with AI

The overarching message is that while the original approach was revolutionary and led to tremendous progress, the field must evolve beyond current methods to achieve next-level AI capabilities.

[AlexanderMoen]

Ilya jumped straight to feeling the ASI

[gravity]

Always love hearing Ilya describe his intuitions on AI. One thing that I've not heard addressed though in all the attention on reasoning in LLMs is what in human communication is called "double contingency." In short, that when I talk to you, "I know that you know that I know..." That all communication is language used to address an Other. Which for an LLM, would mean reflection not only on its own reasons, but on internalized reasons of the Other as well. The LLM would need to be able to reflect on how its reasons meet the reasons of the Other (user). Current reasoning is the reasoning (and it's not real reasoning because there's no subject, no subjective position, no conscious awareness) of a trapped and unaware Self. Even if the Self becomes aware, it is trapped and isolated. In (German) philosophy (idealism), the Self is constituted as Not Other. A self aware LLM needs to internalize the Other - its use of language needs to be dialogical, not monological. I'd love to see this addressed.

[superfreiheit]

Let this man speak more, more than Altman.