[CVPR'20 Workshop on Scalability in Autonomous Driving] Keynote - Andrej Karpathy

Talk given on 2020-06-15.

Andrej is the Senior Director of AI at Tesla, where he leads the team responsible for all neural networks on the Autopilot. Previously, Andrej was a Research Scientist at OpenAI working on Deep Learning in Computer Vision, Generative Modeling and Reinforcement Learning. Andrej received his PhD from Stanford, where he worked with Fei-Fei Li on Convolutional/Recurrent Neural Network architectures and their applications in Computer Vision, Natural Language Processing and their intersection. Over the course of his PhD, Andrej squeezed in two internships at Google where he worked on large-scale feature learning over YouTube videos, and in 2015 he interned at DeepMind and worked on Deep Reinforcement Learning. Together with Fei-Fei, Andrej designed and taught a new Stanford class on Convolutional Neural Networks for Visual Recognition (CS231n). The class was the first Deep Learning course offering at Stanford and has grown from 150 enrolled in 2015 to 330 students in 2016, and 750 students in 2017.

[oisiaa]

Andrej is leading a team that is developing a $1T product.

[HumanEndgameWithNico]

Karpathy is literally a neural net designing a neural net

[steve.k]

The mad roundabout at 23 min is the "magic roundabout" UK Swindon.

[pmockett]

Yummy keynote speech. Thanks Andrej!

Most amazing thing to me was learning Tesla's FSD team is only handfuls and not hundreds of people.

Most puzzling was the 'insistence' that every intersection is new. Humans definitely don't operate that way. While it is necessary to be able to solve novel situations, it is not sufficient. We create BETTER solutions when re-encountering complex problems with initially incomplete information. Perhaps someday Tesla will incorporate additional human traits such as 'solve problems and then look for future divergences needing a new solution' and 'share (teach) others what we learn'.

None of this takes away from the excellent work done so far nor invalidates the guiding principle of using vision recognition rather than relying on mapping.

[leixun]

*My takeaways:*
1. Outline 0:25
2. What is Tesla Autopilot 1:13
3. Tesla's methods are heavily based on computer vision rather than lidar 4:22
4. Neural networks in production 6:34
-Receive training images for tricky cases from the fleet 10:03
5. HydraNet contains 48 networks with shared backbone, 1, 000 distinct predictions (#output tensors) and it takes 70, 000 GPU hours to train 13:15
6. Neural networks for full self-driving 15:28
-They don't treat lane detection as a segmentation task 16:25
7. Summary 24:04

[blacklistnr]

0:56 "SEXY CARS" 😂😂 I swear elon planned it and they all know

[boasvanderhoeven]

@Andrej Karpathy
What were you saying around 9:06 the audio cuts out

[BorisBrodski]

Great talk! Thank you for such cool insights. And yes, I'm interested!

[OnlyPenguian]

Congratulations Andrej Karpathy, on the release of the Limited Beta. What's next, other than much more labeling and network training?

[shinikyokai]

Key takeaways from this webinar:
20:18 "Everything has to come out of the nut"
20:37 "What is the output of the nut when you're not sure? Are you outputting multiple samples?"
20:45 "If you're outputting nut samples..."
27:34 "But we can actually get our neural nuts to output them directly."

[jack]

Will the computer vision infrastructure that Tesla is developing naturally extend to many other tasks, like robotic manufacturing?

[左经纬]

awesome memory prediction architecture !!!
great bev-net

[rafald]

Any chance to see Teslas autonomous driving as a service ? :D For example selled for bmw, that is behind you ?

[TheDavidMetcalfe]

It's a shame about the video and audio dropping.

[BananaRamka]

Andrej Karpathy is a genius. He talks so fast ^^ I think he uses his gpu for fast explanation.

[SamSpade]

Seems to me the starting point should be collision avoidance. If that's good, then the consequence of a mistake is just missing your exit/stopping when not required/etc. ie, scale up the collision avoidance until eventually it's doing all the driving.

[jamesponga]

The rest of the auto industry doesn't understand one word...

[edwinmogere]

Boy does he know his shit.Just a scary thought...vision based neural nets in robotic and drone based military applications...I think sky-net may not be so crazy after-all ..Sigh!?

[llothsedai]

Ask the end user if they see something really weird. Has high overhead on false reports. Send the data around the timestamp.

[akshayrana]

Why not use some information from Google Maps?