Neural and Non-Neural AI, Reasoning, Transformers, and LSTMs

Jürgen Schmidhuber, the father of generative AI shares his groundbreaking work in deep learning and artificial intelligence. In this exclusive interview, he discusses the history of AI, some of his contributions to the field, and his vision for the future of intelligent machines. Schmidhuber offers unique insights into the exponential growth of technology and the potential impact of AI on humanity and the universe.

MLST is sponsored by Brave:

TOC
00:00:00 Intro
00:03:38 Reasoning
00:13:09 Potential AI Breakthroughs Reducing Computation Needs
00:20:39 Memorization vs. Generalization in AI
00:25:19 Approach to the ARC Challenge
00:29:10 Perceptions of Chat GPT and AGI
00:58:45 Abstract Principles of Jurgen's Approach
01:04:17 Analogical Reasoning and Compression
01:05:48 Breakthroughs in 1991: the P, the G, and the T in ChatGPT and Generative AI
01:15:50 Use of LSTM in Language Models by Tech Giants
01:21:08 Neural Network Aspect Ratio Theory
01:26:53 Reinforcement Learning Without Explicit Teachers

Refs:
★ "Annotated History of Modern AI and Deep Learning" (2022 survey by Schmidhuber):
★ Chain Rule For Backward Credit Assignment (Leibniz, 1676)
★ First Neural Net / Linear Regression / Shallow Learning (Gauss & Legendre, circa 1800)
★ First 20th Century Pioneer of Practical AI (Quevedo, 1914)
★ First Recurrent NN (RNN) Architecture (Lenz, Ising, 1920-1925)
★ AI Theory: Fundamental Limitations of Computation and Computation-Based AI (Gödel, 1931-34)
★ Unpublished ideas about evolving RNNs (Turing, 1948)
★ Multilayer Feedforward NN Without Deep Learning (Rosenblatt, 1958)
★ First Published Learning RNNs (Amari and others, ~1972)
★ First Deep Learning (Ivakhnenko & Lapa, 1965)
★ Deep Learning by Stochastic Gradient Descent (Amari, 1967-68)
★ ReLUs (Fukushima, 1969)
★ Backpropagation (Linnainmaa, 1970); precursor (Kelley, 1960)
★ Backpropagation for NNs (Werbos, 1982)
★ First Deep Convolutional NN (Fukushima, 1979); later combined with Backprop (Waibel 1987, Zhang 1988).
★ Metalearning or Learning to Learn (Schmidhuber, 1987)
★ Generative Adversarial Networks / Artificial Curiosity / NN Online Planners (Schmidhuber, Feb 1990; see the G in Generative AI and ChatGPT)
★ NNs Learn to Generate Subgoals and Work on Command (Schmidhuber, April 1990)
★ NNs Learn to Program NNs: Unnormalized Linear Transformer (Schmidhuber, March 1991; see the T in ChatGPT)
★ Deep Learning by Self-Supervised Pre-Training. Distilling NNs (Schmidhuber, April 1991; see the P in ChatGPT)
★ Experiments with Pre-Training; Analysis of Vanishing/Exploding Gradients, Roots of Long Short-Term Memory / Highway Nets / ResNets (Hochreiter, June 1991, further developed 1999-2015 with other students of Schmidhuber)
★ LSTM journal paper (1997, most cited AI paper of the 20th century)
★ xLSTM (Hochreiter, 2024)
★ Reinforcement Learning Prompt Engineer for Abstract Reasoning and Planning (Schmidhuber 2015)
★ Mindstorms in Natural Language-Based Societies of Mind (2023 paper by Schmidhuber's team)
★ Bremermann's physical limit of computation (1982)

EXTERNAL LINKS
CogX 2018 - Professor Juergen Schmidhuber
Discovering Neural Nets with Low Kolmogorov Complexity and High Generalization Capability (Neural Networks, 1997)
The paradox at the heart of mathematics: Gödel's Incompleteness Theorem - Marcus du Sautoy
The Philosophy of Science - Hilary Putnam & Bryan Magee (1977)
Optimal Ordered Problem Solver
Levin's Universal Search from 1973
On Learning to Think
Mindstorms in Natural Language-Based Societies of Mind

Untersuchungen zu dynamischen neuronalen Netzen
Evolutionary Principles in Self-Referential Learning
Hans-Joachim Bremermann
Highway Networks
The principles of Deep Learning Theory
Discovering Problem Solutions with Low Kolmogorov Complexity and High Generalization Capability (ICML 1995)
"History of Modern AI and Deep Learning":

[kilianlindberg]

A gentle reminder at 59:57: We're all inherently capable of complex calculations, even if we can't explicitly articulate or write them down on paper. For instance, when a toddler grows into a kid and catches an apple, they're unconsciously calculating its trajectory and acceleration. This innate ability is a testament to our brain's capacity to process and understand physics, even without formal mathematical training. Just a humble reminder that may be .. fruitful.

[Renvoxan]

Hell yeah, we are all getting Schmidhubered today 😍

He is the 🐐

[johnk]

A dialogue between Schmidhuber and Wolfram or Chaitin or Friston on algorithmic information theory, computational irreducibility, free energy principle and learning would be interesting and insightful.

[oncedidactic]

“Deep learning is all about depth.”
Surprisingly deep statement.

[badrraitabcas]

Didn't watch yet, but finally someone who talks ai, not science fiction

[MattGarcia]

Jurgen talks in the simplest plain English — a hallmark of intelligence

[rcpinto]

26:24
Interviewer: **mentions something someone said on Twitter**
Schmidhuber: Yeah, my mom said that in the 70's.

That's a whole new level of Schmidhubering!!

[markonjegomir]

MLST is just a permutation of LSTM. We got Schmidhuberd, and no one noticed. He can't keep getting away with this.

[MartinLaskowski]

'cos this is gold. Thank you Tim and crew.

[loofatar]

He is quite an interesting character, with solid vision. He should come on media more regularly.

[proximo]

I love Schmidhuber's argument on practical computation machine. I think many CS people don't get it

[mohamedfouad]

WOW !!! he drops heavy wisdom so casually!

[MartinLaskowski]

"Unlimited and infinite are two different things" ... is absolutely bloody right!

[bfrr]

There are infinitely many Python programs no particular build of its interpreter can execute, and only finitely many it can execute, for the simple fact that there are architectural limitations in hardware and software, like the wideness of the internal source file offset counter. Even yet, before reaching that limit, there's the file system limitation on the size of files, irregardless of storage space. There are limits everywhere in any concrete hardware and software implementation of anything.

[mfpears]

I added this to my Listen Later playlist so fast. Jurgen is the best

[alexanderbrown-dgsy]

Wow. The OG. The Don. Doctor I really did it first. Big daddy. No diddy. Such a fan. I was literally just re-reading his self-referential weight paper. I really hope he expands on that line of research, maybe improve upon test-time training layers.

One more nickname for the legend. Mr I can actually knock out all the opps 😂. Why is he so buff 😂. Cheers Doc.

Jokes aside, the greatest mind in modern deep learning. I truly enjoyed this conversation.

[pauloabelha]

26:35 Even Mom Schmidhuber said it first about AI washing the dishes. This family is 🔥
Schmidhuber is a legend!

[JD-jlyy]

So far the scaling laws crowd has been right and caught the rest of the ML community off guard, but curious to see who turns out to be right going forward!

[CodexPermutatio]

Schmidhuber! So great to see him on this channel.

[Daniel-Six]

Whatever the conflicts about academic priority, it's pretty clear that Schmidhuber is a deeper thinker than LeCun, Hinton, Suleyman and their crowd.