comma ai | Learning a Driving Simulator | Yassine Yousfi | COMMA_CON talks | Research | ML

Yassine Yousfi social media:

Research:

View original video on Youtube:

Chapters:
00:00 intro
00:30 learning a driving simulator
00:45 imagined, predicted videos via ML
01:49 why a simulator
02:58 why not a classical simulator
04:05 small offset simulator
06:35 learning a driving simulator paper
07:13 ML simulator architecture
07:40 image tokenizer
11:15 dynamics transformer
14:05 it works, straight, turning right, left
15:03 accelerating, braking
15:30 commaVQ opensource dataset
16:29 side note 1: tokenized driving models
17:30 side note 2: smooth decoder
18:47 side note 3: loss function for driving in ML simulator
20:12 next steps, auto-regressive sampling
22:05 using the learned simulator for traning driving models
22:15 beautiful rollouts properties
23:49 questions
24:30 conditioning on language?
25:44 bottlenecks to inference for real time
26:57 aplications for human indistinguishable rollouts
27:49 model collapse from simulation data
29:12 mass auto encoders
30:12 video decoder, INB frame, general iframe and p frame
31:15 output from the simulator used for training policy models
31:37 two models, end to end, same loss function
32:45 policy model will be trained on the simulator, not the vision model
32:58 how to keep up with machine learning research
33:52 you don't want to be the first one to replicate a paper
34:23 state of the art, leveraging other work and open sourcing
35:09 where do the model names come from, nicki minaj, Nicholas Cage
35:30 tokenize because cross entropy is the best loss function
36:28 interpreting the tokens
37:23 conditioning on geolocation
38:10 scaling to any type of robotics problem
39:17 adding language as the first input
40:43 why does it flicker
42:05 smoothing decoder, RNN layer

Official communication channels:

How to get a job:
How to collaborate:

Are you interested in openpilot?
Knowledge base:
Check out the code:
Is my car supported?
Frequently Asked Questions:
How to setup openpilot:
Comma Secure Shell:
API Documentation:
CAN analysis tool:
Review and annotate your driving data:
Leaderboard:
Comma Connect App:
Research:

Official George Hotz communication channels:

Follow for notifications:
Unofficial communities and resources:

Thank you for reading and using the SHOW MORE button.
We hope you enjoy watching George's videos as much as we do.
See you at the next video.

[geohotarchive]

High quality version uploaded -> youtu.be/-KMdo9AWJaQ | from $1250 buy -> comma 3X comma.ai/shop/comma-3x | If you have skill apply on -> comma.ai/jobs | Join the community on github.com/commaai/openpilot | Check if your car is supported -> comma.ai/vehicles | How to setup your car comma.ai/setup | follow the official youtube.com/commaai for more livestreams | Stay up to date by following twitter.com/comma_ai | Research:
Chapters:
00:00 intro
00:30 learning a driving simulator
00:45 imagined, predicted videos via ML
01:49 why a simulator
02:58 why not a classical simulator
04:05 small offset simulator
06:35 learning a driving simulator paper
07:13 ML simulator architecture
07:40 image tokenizer
10:30 pose tokenizer (np.digitize)
11:15 dynamics transformer
14:05 it works, straight, turning right, left
15:03 accelerating, braking
15:30 commaVQ opensource dataset
16:29 side note 1: tokenized driving models
17:30 side note 2: smooth decoder
18:47 side note 3: loss function for driving in ML simulator
20:12 next steps, auto-regressive sampling
22:05 using the learned simulator for traning driving models
22:15 beautiful rollouts properties
23:49 questions
24:30 conditioning on language?
25:44 bottlenecks to inference for real time
26:57 aplications for human indistinguishable rollouts
27:49 model collapse from simulation data
29:12 mass auto encoders
30:12 video decoder, INB frame, general iframe and p frame
31:15 output from the simulator used for training policy models
31:37 two models, end to end, same loss function
32:45 policy model will be trained on the simulator, not the vision model
32:58 how to keep up with machine learning research
33:52 you don't want to be the first one to replicate a paper
34:23 state of the art, leveraging other work and open sourcing
35:09 where do the model names come from, nicki minaj, Nicholas Cage
35:30 tokenize because cross entropy is the best loss function
36:28 interpreting the tokens
37:23 conditioning on geolocation
38:10 scaling to any type of robotics problem
39:17 adding language as the first input
40:43 why does it flicker
42:05 smoothing decoder, RNN layer

[itsyaccine]

Happy to see a fellow moroccan 🇲🇦 at Comma.

[johanngerberding]

comma is so underrated, jesus! keep up the amazing work!

[antikras]

That sounds like Dreamer by Haffner et all but with transformers. Cool.

[ultrasound]

35:11 Bruh..GeoHotz is definitely behind that idea 🤣