What is interesting to an AI agent?

AI professor Jeff Clune ruminates on open-ended evolutionary algorithms—systems designed to generate novel and interesting outcomes forever. Drawing inspiration from nature’s boundless creativity, Clune and his collaborators aim to build “Darwin Complete” search spaces, where any computable environment can be simulated. By harnessing the power of large language models and reinforcement learning, these AI agents continuously develop new skills, explore uncharted domains, and even cooperate with one another in complex tasks.

SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events?

They are hosting an event in Zurich on January 9th with the ARChitects, join if you can.

***

A central theme throughout Clune’s work is “interestingness”: an elusive quality that nudges AI agents toward genuinely original discoveries. Rather than rely on narrowly defined metrics—which often fail due to Goodhart’s Law—Clune employs language models to serve as proxies for human judgment. In doing so, he ensures that “interesting” always reflects authentic novelty, opening the door to unending innovation.

Yet with these extraordinary possibilities come equally significant risks. Clune says we need AI safety measures—particularly as the technology matures into powerful, open-ended forms. Potential pitfalls include agents inadvertently causing harm or malicious actors subverting AI’s capabilities for destructive ends. To mitigate this, Clune advocates for prudent governance involving democratic coalitions, regulation of cutting-edge models, and global alignment protocols.

Jeff Clune:

(Interviewer: Tim Scarfe)

TOC:
1. Introduction
[00:00:00] 1.1 Overview and Opening Thoughts

2. Sponsorship
[00:03:00] 2.1 TufaAI Labs and CentML

3. Evolutionary AI Foundations
[00:04:12] 3.1 Open-Ended Algorithm Development and Abstraction Approaches
[00:07:56] 3.2 Novel Intelligence Forms and Serendipitous Discovery
[00:11:46] 3.3 Frontier Models and the 'Interestingness' Problem
[00:30:36] 3.4 Darwin Complete Systems and Evolutionary Search Spaces

4. System Architecture and Learning
[00:37:35] 4.1 Code Generation vs Neural Networks Comparison
[00:41:04] 4.2 Thought Cloning and Behavioral Learning Systems
[00:47:00] 4.3 Language Emergence in AI Systems
[00:50:23] 4.4 AI Interpretability and Safety Monitoring Techniques

5. AI Safety and Governance
[00:53:56] 5.1 Language Model Consistency and Belief Systems
[00:57:00] 5.2 AI Safety Challenges and Alignment Limitations
[01:02:07] 5.3 Open Source AI Development and Value Alignment
[01:08:19] 5.4 Global AI Governance and Development Control

6. Advanced AI Systems and Evolution
[01:16:55] 6.1 Agent Systems and Performance Evaluation
[01:22:45] 6.2 Continuous Learning Challenges and In-Context Solutions
[01:26:46] 6.3 Evolution Algorithms and Environment Generation
[01:35:36] 6.4 Evolutionary Biology Insights and Experiments
[01:48:08] 6.5 Personal Journey from Philosophy to AI Research

Shownotes:
We craft detailed show notes for each episode with high quality transcript and references and best parts bolded.

CORE REFS:

[00:02:35] POET: Generating/solving complex challenges | Wang, Lehman, Clune, Stanley

[00:11:10] Why Greatness Cannot Be Planned | Stanley, Lehman

[00:17:05] Automated capability discovery in foundation models | Lu, Hu, Clune

[00:18:10] NEAT: NeuroEvolution of Augmenting Topologies | Stanley, Miikkulainen

[00:26:50] Novelty search vs objective-based optimization | Lehman, Stanley

[00:28:55] AI-generating algorithms approach to AGI | Jeff Clune

[00:41:10] Learning Minecraft from human gameplay videos (VPT) | Baker, Akkaya et al.

[00:44:00] Thought Cloning: Imitating human thinking | Hu, Clune

[01:15:10] Automated Design of Agentic Systems (ADAS) | Hu, Lu, Clune

[01:32:30] OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness | Faldor, Zhang, Cully, Clune

[MachineLearningStreetTalk]

Sorry folks - Just noticed there are a couple of visual references in the wrong place in the first 20 minutes, sorry - not worth pulling the video over though.

[___Truth___]

After seeing so many intellectuals & academics on this channel, I’ve come to only really appreciate those that are involved in making their ideas concrete, actionable, & executable. Otherwise it seems like ALL OF THESE PEOPLE seem to be stuck in _Conceptual Hell_ . Never breaking out of the things trapping them inside their minds.

[Aaron-tlzy]

Having just read why greatness can't be planned which sparked an interest in evolutionary algorithms, this episode was greatly appreciated

[MachineLearningStreetTalk]

[00:02:35] Essay on AI benefits and safety | Dario Amodei
[00:04:50] POET algorithm for generating and solving complex challenges | Wang, Lehman, Clune, Stanley
[00:07:20] Blue Brain Project: molecular-level brain simulation | Henry Markram
[00:08:05] DreamCoder: Wake-sleep Bayesian program learning | Kevin Ellis et al.
[00:08:35] Computational models of human cognition | Joshua B. Tenenbaum
[00:11:10] Why Greatness Cannot Be Planned | Stanley, Lehman
[00:14:00] Goodhart's Law on metrics as targets | Charles Goodhart
[00:17:05] Automated capability discovery in foundation models | Lu, Hu, Clune
[00:18:10] NEAT: NeuroEvolution of Augmenting Topologies | Stanley, Miikkulainen
[00:21:10] The grokking phenomenon in neural networks | Power et al.
[00:26:50] Novelty search vs objective-based optimization | Lehman, Stanley
[00:27:35] "I know it when I see it" obscenity case | Justice Potter Stewart
[00:28:55] AI-generating algorithms approach to AGI | Jeff Clune
[00:30:40] The invisible hand economic principle | Adam Smith
[01:32:30] OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code | Faldor, M., Zhang, J., Cully, A., & Clune, J.
[00:36:40] Genie: Neural network world simulation | Bruce et al.
[00:36:45] Genie 2: Large-scale foundation world model | Parker-Holder et al.
[00:38:05] Inductive vs transductive AI reasoning | Kevin Ellis et al.
[00:38:45] Thinking, Fast and Slow | Daniel Kahneman
[00:41:10] Learning Minecraft from human gameplay videos | Baker, Akkaya et al.
[00:44:00] Thought Cloning: Imitating human thinking | Hu, Clune
[00:47:15] The Language Game: Origins of language | Christiansen, Chater
[00:48:45] Facebook AI language creation fact check | USA Today
[00:54:20] The Mind Is Flat: Improvising brain theory | Nick Chater
[00:57:50] Constitutional AI methodology | Bai et al.
[01:04:50] Managing extreme AI risks | Bengio, Clune et al.
[01:10:25] US Executive Order on AI regulation | The White House
[01:15:10] Automated Design of Agentic Systems | Hu, Lu, Clune
[01:20:30] The Lottery Ticket Hypothesis | Frankle, Carbin
[01:24:15] In-context learning in language models | Dong et al.
[01:25:40] Meta-learning for exploration problems | Norman, Clune et al.
[01:36:25] Replaying the tape of life | Stephen Jay Gould
[01:37:05] Long-Term E. coli Evolution Experiment | Richard E. Lenski
[01:41:50] Carcinization patterns in crabs | Luque et al.
[01:50:35] Evolutionary robotics and 3D printing | Hod Lipson
[01:56:50] NEAT: Evolving neural networks | Stanley, Miikkulainen

[dr.mikeybee]

Exploration is paradoxical. In many cases, reducing the solution set requires application of bias. In inference, we are always trying to minimize bias. In exploration, in order to optimize time, we need to apply bias. Evolution has endowed us with neural circuitry that act as kernels that provide helpful abstract representations and helpful biases.

[tylermoore]

These topics are so deep and so formidable, going to the very core of reality and existence, that I am flabbergasted at how unfazed Jeff seems taking them on. Evolution is playing with a very large library of materials, the periodic table and many forces and fields, all of whose properties we still do not understand. The idea that we can "abstract" away all of this in a toy world on a computer is...I don't know, pretty out there. Also, I am not sure how this cosmic project of figuring out open-ended creativity intersects with the practical, task-focused AI that everyone else is working towards.

[snarkyboojum]

Great! Human creativity is still so poorly understood, yet it’s likely a core part of human intelligence. It’s wonderful to hear from people like Jeff Clune. Thank you for doing this, MLST. The number of ideas you’ve captured over the past couple of years is incredibly valuable. I appreciate all the hard work and the effort to make these fascinating ideas broadly accessible.

Listening to the conversation, one point that didn’t convince me was the idea of relying on a large language model to tell us what’s interesting. While LLMs might show what humans have historically found interesting, I’m not sure they have a strong forward-looking capacity to determine whether something truly new is interesting. I also doubt they can “know it when they see it, ” to use the example mentioned in the conversation. Even if they could learn historical patterns for what made past ideas ‘interesting, ’ that doesn’t necessarily mean they could recognize something genuinely new as interesting. To me, that’s the missing piece, and perhaps it’s central to human creativity.

[GrindThisGame]

What a wonderful talk. It's nice to see UBC represented here. Jeff mentions that safety measures would likely slow progress but that it is worth slowing down. He also says that if "we" don't develop ASI someone else will. I'm guessing he means a non US/Canada nation? But if another non "we" country or entity develops ASI without safety then they will go faster and likely get ASI first. All that said, it seems futile to try to control an ASI.

[TBOBrightonandHove]

Excellent just excellent guest and conversation. Lots for future exploration 😀

[earleyelisha]

Added this one to my fav MLST episodes!

[lionmanking]

Where and when will be the tufa event?

[swayson]

ah finally! Love this topic and Jeff Clune et al, has legendary research on this front.

[XAirForcedotcom]

It’s a lot easier to understand that we’re alive when you it’s suggested that the thing we are in is also alive just like we have bacteria inside of us

[cezarambros]

What if MLST was a social company like a reserch lab ? would we get to AGI faster? why isn't it a Reserch company what is it missing?

[michaelwoodby]

Going to need constant retraining or huge context windows to take advantage of the iterative AI research paper idea. In the current form they're going to be able to make exactly one leap beyond what humans have done, as came up in the interview. Then, rather than lapping human progress with the increased speed, they'll have to wait for humans to catch up and add the new work to the next set of cumulative data.
Getting one research paper a week early isn't going to cause accelerated progress until the following paper can take advantage of that extra week.
Edit: Dang, got towards the end of the video and that's the whole discussion.

[wwkk]

Nice to have you posting again! Would love to hear your thoughts on the LLMs are glorified databases viewpoint from last year. Love the show 🎉

[GrindThisGame]

We have continuous models on each new version of a frontier model but the delay is larger vs. how humans are continually learning. So in theory they should be as capable if training times compress with higher compute.

[heterotic]

There is a hierarchy of 'thoughts', actions can map to conscious and subconscious network states. The action is the dual of the state (the whole state). The domain mastery discussed here seems to map to conscious network states, whose determinant is focus. Subconscious network states do not translate well to language spaces, but may be inferred as the distance between focused network states and actions (network updating)..

[juan_ta]

15:10 Some time ago I saw a not-into-the-detail but still suggestive video (here in yt, I wonder if I can recall the channel) which brought with the idea of "understanding" (compression of available knowledge through new fitting patterns, new data providing the clues to them) as a goal for a model to pursue.

[Amon-eycy]

A most exceptional conversation and scientist.