No Priors Ep. 80 | With Andrej Karpathy from OpenAI and Tesla

Andrej Karpathy joins Sarah and Elad in this week of No Priors. Andrej, who was a founding team member of OpenAI and the former Tesla Autopilot leader, needs no introduction. In this episode, Andrej discusses the evolution of self-driving cars, comparing Tesla's and Waymo’s approaches, and the technical challenges ahead. They also cover Tesla’s Optimus humanoid robot, the bottlenecks of AI development today, and how AI capabilities could be further integrated with human cognition. Andrej shares more about his new mission Eureka Labs and his insights into AI-driven education and what young people should study to prepare for the reality ahead.

Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @Karpathy

Show Notes:
0:00 Introduction
0:33 Evolution of self-driving cars
2:23 The Tesla vs. Waymo approach to self-driving
6:32 Training Optimus with automotive models
10:26 Reasoning behind the humanoid form factor
13:22 Existing challenges in robotics
16:12 Bottlenecks of AI progress
20:27 Parallels between human cognition and AI models
22:12 Merging human cognition with AI capabilities
27:10 Building high performance small models
30:33 Andrej’s current work in AI-enabled education
36:17 How AI-driven education reshapes knowledge networks and status
41:26 Eureka Labs
42:25 What young people study to prepare for the future

[Nkunzi]

I generally listen to podcasts at 1.5 speed. Andrej is one of the few people whom I have to play at normal speed to keep up with.

[LeonidKotelnikov-jgfi]

I had to check my playback speed settings, because I thought I’m on x1.5

[motobanite]

Karpathy is a truly great human being.

[niksapraljak]

IMO... Listening to Andrej Karpathy talk about innovation, work on deep learning research (e.g. assembling LayerNorms, Skip connections, and other parts to create a trainable "tissue" called transformers) and current/future problems is a breath of fresh air for young researchers like myself.

[danklebanov]

Andrej is amazing. On the topic of education, he is proposing that current technology would allow for AI to be the front-end to the "perfect" course, where a human designs the course, and uses AI to scale themselves. One critical piece, especially for younger learners, is the human connection. Some kids (like the young Andrej's of this world) can be entirely self-driven, but many need to be inspired and guided by another person. I wonder how AI can be used to "super-charge" a teacher, rather than replacing the human at the front.

[ZacheryGlass]

This guy is an actual genius. You can tell just by the expresses his ideas and is so forward thinking

[hsiaohu]

thank you Andrej for explaining technical things that a layperson such as myself can understand, or at least get a sense of😃

[gazmendqose]

Just discovered this podcast. What a gem . Now I need to catch up with 80+ episodes 😆

[SLAM]

great stuff! Glad to see Andrej very happy doing his own things rather than being limited at OpenAI

[LoisSharbel]

Joy! By accident, I stumbled onto this podcast and it's brilliant! Andre Karpathy is so likeable, one wants to concentrate to catch the meaning as he speeds through his ideas. He impresses me as a good person and good for the world. Thank you for this!!! I'm a follower, not with understanding of physics but in getting a peek at the edges.

[kunstrikerasochi]

As always with Andrej, awesome insights

[muhannadobeidat]

Andrej is a legend! He answers normal questions by saying “this will take you down to a local minimum”. This should now be an expression in the English language.
Great interview, thanks for putting this together and letting Andrej share more gems with the rest of us.

[XiallaLife]

Best interview ever relating to AI, FSD and Optimus.

[righttiming]

legendary guest

most underrated new pod in the world imo.

get ilya, elon, geohot, friend founder, sama next 💙

[TheMrPopper]

I think Andrej will be written into history books for years to come, he’s taught so many and brought open source to the forefront of technological innovation

[ThysRoes]

Yes we need technical people, we also still need storytellers, educators, performers and philosophers to give meaning to what we can create with technology.

[SamsonMathew]

🎯 Key points for quick navigation:

00:02:00 *🚗 Andrej Karpathy discusses his work in self-driving technology and its parallels with AGI, emphasizing the impressive strides made over the past decade.*
00:03:21 *🤖 Karpathy believes Tesla is ahead of Waymo in self-driving capabilities, attributing Tesla's strength to its scalable deployment of vehicles.*
00:04:03 *🌐 Tesla's approach to sensors entails extensive use of expensive equipment during training rather than in operational deployment, making it a cost-effective strategy.*
00:06:23 *🏭 Karpathy argues Tesla should be viewed as a robotics company rather than just a car manufacturer due to its innovative approach in robotics at scale.*
00:08:45 *🔄 He suggests that the best early applications for humanoid robots will likely emerge in businesses, focusing on tasks like material handling to minimize legal liabilities.*
00:10:37 *🌱 The promise of humanoid robots includes niche applications like leaf blowing, showcasing the practicality of their development.*
00:11:36 *🧠 Karpathy emphasizes the importance of transfer learning between tasks, advocating for a single humanoid platform to optimize efficiency.*
00:15:22 *✨ He highlights the transformative impact of Transformers in neural network architecture, marking a significant leap in research capabilities and scalability.*
00:16:33 *🔍 Karpathy concludes that the bottlenecks in AI research have shifted from architecture to loss functions and datasets.*
17:15 *📊 Data innovation focus is shifting towards synthetic data and enhanced data collection methods due to limitations in available internet data.*
17:57 *🧠 Understanding the brain's reasoning processes could advance AI development, as current models help refine future designs.*
19:10 *⚠️ Ensuring diversity and richness in synthetic data is crucial to prevent model collapse and maintain data quality.*
21:16 *🧬 Transformers may eventually outperform human cognitive abilities in specific tasks but are currently constrained by data limitations.*
22:40 *🤖 Human-AI augmentation is likely to evolve, blurring the lines between technology and human cognition.*
25:30 *🔄 The ecosystem of large AI labs may create a reliance on closed platforms, raising concerns about ownership and control of cognitive enhancements.*
27:09 *📏 Smaller, efficient AI models could be viable, focusing on cognitive core functionalities rather than unnecessary data retention.*
29:55 *🌍 AI education tools could democratize learning, empowering individuals with personalized and scalable access to knowledge.*
32:06 *🎓 AI has the potential to revolutionize education by serving as a personalized interface that adapts to students' needs, enhancing their learning experience.*
34:26 *🏆 Better educational tools and curricula could significantly enhance individual performance, similar to advancements in sports training methods.*
34:54 *🌍 Language adaptability in AI translations is effective, serving as low-hanging fruit for improving education.*
35:50 *🔍 Personal learning context varies; AI should adapt to individual backgrounds and learning rates for effectiveness.*
36:44 *🏛️ Lineage in the AI research community influences knowledge propagation, but it's crucial to minimize its gatekeeping effect.*
37:12 *🤝 Community clustering in AI facilitates knowledge sharing and education, shaping personal motivations and cultural values.*
39:06 *🎓 Education should become more accessible, helping people learn for practical reasons while also shifting towards personal enjoyment.*
40:04 *🧠 Learning should be viewed as mental exercise; the challenge of education can parallel the effort needed in physical fitness.*
41:29 *👩‍🎓 The upcoming course targets undergraduates in technical fields, emphasizing continuous learning as technology evolves.*
42:26 *📚 Young learners should focus on math, physics, and computer science as foundational skills for effective problem solving and adaptability in the future.*

[TiagoChilanti]

Andrej seems to speak in 1.75X
I usually whatch videos on 2X or 1.75X but this one I had to watch at 1X

[nickmerrill]

42:33 I think Karpathy’s points about what learners should focus on are excellent. I especially appreciate his openness to 20% non-“symbol manipulation” subjects.

The only thing I would add is that subjects like writing and history can also strengthen your reasoning abilities. I agree that memorizing facts is not a good use of a learner’s time, but reasoning about cause and effect is. Such reasoning can be practiced outside of “STEM”.

I think a lot of folks with STEM background (including myself) have a slight bias against non-STEM subjects, so I feel it’s important to add that perspective.

Finally, there are some important “human skills” such as empathy that can arise from subjects like literature, so I appreciate Karpathy’s openness to being well rounded.

Thanks for the solid interview!

[ssumargroup]

AI being the Front End for a backend heavily technical course. That´s it. I am home now. Thanks GOAT.