comma ai | Learning a Driving Simulator | Yassine Yousfi | COMMA_CON talks | Research | HQ version

Date of the stream 29 Jul 2023.

Yassine Yousfi social media:

Research:

View original video on Youtube:

Chapters:
00:00 intro
00:20 learning a driving simulator
00:36 imagined, predicted videos via ML
01:44 why a simulator
02:50 why not a classical simulator
04:00 small offset simulator
06:25 learning a driving simulator paper
07:10 ML simulator architecture
07:25 image tokenizer
08:32 source - 5120 bits - 1280 bits
11:06 dynamics transformer
14:00 it works, straight, turning right, left
14:56 accelerating, braking
15:25 commaVQ opensource dataset
15:50 2x$1000 bounties
16:50 side note 1: tokenized driving models
17:25 side note 2: smooth decoder
18:29 raw - smooth rollout
18:44 side note 3: loss function for driving in ML simulator
20:02 next steps, auto-regressive sampling
22:03 using the learned simulator for traning driving models
22:13 beautiful rollouts properties
23:40 questions
23:52 conditioning on language?
24:10 pose tokens
25:18 bottlenecks to inference for real time
26:23 aplications for human indistinguishable rollouts
27:13 model collapse from simulation data
28:35 mass auto encoders
29:40 video decoder
30:00 INB frame, general i frame and p frame
30:40 output from the simulator used for training policy models
31:00 two models, end to end, same loss function
32:08 policy model will be trained on the simulator, not the vision model
32:25 how to keep up with machine learning research
33:15 you don't want to be the first one to replicate a paper
33:48 state of the art, leveraging other work and open sourcing
34:35 where do the model names come from, Nicki Minaj, Nicholas Cage
34:55 tokenize because cross entropy is the best loss function
35:50 interpreting the tokens
36:45 conditioning on geolocation
37:35 scaling to any type of robotics problem
38:41 adding language as the first input
40:05 why does it flicker
41:30 smoothing decoder, RNN layer

Official communication channels:

How to get a job:
How to collaborate:

Are you interested in openpilot?
Knowledge base:
Check out the code:
Is my car supported?
Frequently Asked Questions:
How to setup openpilot:
Comma Secure Shell:
API Documentation:
CAN analysis tool:
Review and annotate your driving data:
Leaderboard:
Comma Connect App:
Research:

Official George Hotz communication channels:

Follow for notifications:
Unofficial communities and resources:

Thank you for reading and using the SHOW MORE button.
We hope you enjoy watching George's videos as much as we do.
See you at the next video.

[geohotarchive]

Thank you to the comma team for this amazing event, sharing the livestream and high-quality videos. All credits to them. Follow their official youtube.com/commaai for more livestreams.
Check out github.com/commaai/commavq | github.com/commaai/comma2k19 | github.com/commaai/comma10k | blog.comma.ai
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | openpilot.comma.ai is the best ADAS system in the world | If you have skill apply on -> comma.ai/jobs | Join the community on github.com/commaai/openpilot & discord.comma.ai | Check if your car is supported -> comma.ai/vehicles | How to setup your car comma.ai/setup | Stay up to date by following twitter.com/comma_ai | Support comma.ai at comma.ai/shop
Chapters:
00:00 intro
00:20 learning a driving simulator
00:36 imagined, predicted videos via ML
01:44 why a simulator
02:50 why not a classical simulator
04:00 small offset simulator
06:25 learning a driving simulator paper
07:10 ML simulator architecture
07:25 image tokenizer
08:32 source - 5120 bits - 1280 bits
10:25 pose tokenizer (np.digitize)
11:06 dynamics transformer
14:00 it works, straight, turning right, left
14:56 accelerating, braking
15:25 commaVQ opensource dataset
15:50 2x$1000 bounties
16:50 side note 1: tokenized driving models
17:25 side note 2: smooth decoder
18:29 raw - smooth rollout
18:44 side note 3: loss function for driving in ML simulator
20:02 next steps, auto-regressive sampling
22:03 using the learned simulator for traning driving models
22:13 beautiful rollouts properties
23:40 questions
23:52 conditioning on language?
24:10 pose tokens
25:18 bottlenecks to inference for real time
26:23 aplications for human indistinguishable rollouts
27:13 model collapse from simulation data
28:35 mass auto encoders
29:40 video decoder
30:00 INB frame, general i frame and p frame
30:40 output from the simulator used for training policy models
31:00 two models, end to end, same loss function
32:08 policy model will be trained on the simulator, not the vision model
32:25 how to keep up with machine learning research
33:15 you don't want to be the first one to replicate a paper
33:48 state of the art, leveraging other work and open sourcing
34:35 where do the model names come from, Nicki Minaj, Nicholas Cage
34:55 tokenize because cross entropy is the best loss function
35:50 interpreting the tokens
36:45 conditioning on geolocation
37:35 scaling to any type of robotics problem
38:41 adding language as the first input
40:05 why does it flicker
41:30 smoothing decoder, RNN layer

[KhaosVFX]

Great timing, Harald (Comma CTO) just published today that they finally got a working model fully trained on this simulator on Twitter!

[blaclee]

Wonderful presentation, Thank you Yassine and thank you Camma! Keep it up 🙌

[Ella_]

I truly expect this project to completely revolutionize self driving car

[dev_navdeep]

great presentation, well explained, been working on generating synthetic data, this seems really excited.

[MrFlawor]

Aweseome to watch! 13:00 "In theory" triggers my siri haha

[ArtOfTheProblem]

Amazing work. So is the path ahead is a Learning a Universal simulator, that's "the world model" problem everyone is after?

[-mwolf]

I think the caption for "mass autoencoder" is supposed to be "masked autoencoder" :)

[nishantnikhil]

What's the smooth decoder? Is there a paper?

[avchor]

Will Comma ever transition to making self-driving possible for semi-trucks?