comma ai | comma supercomputer | Greg Hogan | COMMA_CON talks | Machine Learning Training & Compute

Date of the stream 29 Jul 2023.

Greg Hogan social media:

Datacenter:

View original video on Youtube:

Chapters:
00:00 intro
00:28 old office
01:37 not enough power, airflow
02:50 new office - empty room
03:03 adding power, 800A, $125k for power
03:27 walls, aisle for air, racks
03:43 knocking out windows
04:03 big fans for free cooling
04:24 recirculating hot air
04:53 strip curtain
05:10 destratification fans
05:35 end result
06:09 climate control, fans only
06:40 used servers temperature history
07:13 power usage for climate control
07:50 8 kW total at full power for fans
08:40 vfd controller, 80A startup for fans
09:50 control policy for climate control
10:10 humidity target and control
12:55 the new supercomputer
14:00 comparable performance to other supercomputers
15:20 high level specs for the supercomputer
15:51 trainer servers
17:04 rollout servers
18:20 driving data, distributed file system, minikeyvalue
19:06 data diversity
20:07 training openpilot models
21:25 model overview
22:05 vison model
22:35 policy model
23:00 the future, tinygrad
23:40 tinyboxes
23:50 Q & A
24:20 high temperature datacenter
25:44 Greg is awesome
25:54 data center word usage, big fans, cost analysis
27:15 backup solution for power outage
27:25 production customer facing is azure, on-premise for research
27:48 cheap because we accept some downtime
28:50 question for research team
29:10 what GPU is used for training, most dollars per flop
29:40 GPU for rollout servers highest memory bandwith per flop
30:10 question for research team
30:36 time for the whole training process
31:16 too hot for people in the hot datacenter
32:14 lower all-in cost per petaflow for training
33:20 die temperature in hot datacenter
34:15 time to boot up the supercomputer
35:20 fan speeds, exhaust and intake fans at different speeds
36:38 data sales for $200k offer did anyone buy it?

Official communication channels:

How to get a job:
How to collaborate:

Are you interested in openpilot?
Knowledge base:
Check out the code:
Is my car supported?
Frequently Asked Questions:
How to setup openpilot:
Comma Secure Shell:
API Documentation:
CAN analysis tool:
Review and annotate your driving data:
Leaderboard:
Comma Connect App:
Research:

Official George Hotz communication channels:

Follow for notifications:
Unofficial communities and resources:

Thank you for reading and using the SHOW MORE button.
We hope you enjoy watching George's videos as much as we do.
See you at the next video.

[geohotarchive]

Thank you to the comma team for this amazing event, sharing the livestream and high-quality videos. All credits to them. Follow their official youtube.com/commaai for more livestreams.
Check out blog.comma.ai/dev-speed/ | | | comma bounties
from $1250 buy -> comma 3X comma.ai/shop/comma-3x | openpilot.comma.ai is the best ADAS system in the world | If you have skill apply on -> comma.ai/jobs | Join the community on github.com/commaai/openpilot & discord.comma.ai | Check if your car is supported -> comma.ai/vehicles | How to setup your car comma.ai/setup | Stay up to date by following twitter.com/comma_ai | Support comma.ai at comma.ai/shop
Chapters:
00:00 intro
00:28 old office
01:37 not enough power, airflow
02:50 new office - empty room
03:03 adding power, 800A, $125k for power
03:27 walls, aisle for air, racks
03:43 knocking out windows
04:03 big fans for free cooling
04:24 recirculating hot air
04:53 strip curtain
05:10 destratification fans
05:35 end result
06:09 climate control, fans only
06:40 used servers temperature history
07:13 power usage for climate control
07:50 8 kW total at full power for fans
08:40 vfd controller, 80A startup for fans
09:50 control policy for climate control
10:10 humidity target and control
12:55 the new supercomputer
14:00 comparable performance to other supercomputers
15:20 high level specs for the supercomputer
15:51 trainer servers
17:04 rollout servers
18:20 driving data, distributed file system, minikeyvalue
19:06 data diversity
20:07 training openpilot models
21:25 model overview
22:05 vison model
22:35 policy model
23:00 the future, tinygrad
23:40 tinyboxes
23:50 Q & A
24:20 high temperature datacenter
25:44 Greg is awesome
25:54 data center word usage, big fans, cost analysis
27:15 backup solution for power outage
27:25 production customer facing is azure, on-premise for research
27:48 cheap because we accept some downtime
28:50 question for research team
29:10 what GPU is used for training, most dollars per flop
29:40 GPU for rollout servers highest memory bandwith per flop
30:10 question for research team
30:36 time for the whole training process
31:16 too hot for people in the hot datacenter
32:14 lower all-in cost per petaflow for training
33:20 die temperature in hot datacenter
34:15 time to boot up the supercomputer
35:20 fan speeds, exhaust and intake fans at different speeds
36:38 data sales for $200k offer did anyone buy it?

[ps]

Is comma going to adopt liquid neural network for the policy model ?

[alex-dnto]

I wish GH did like a podcast or just regular YouTube videos

[shimondoodkin]

did you undervolted the gpus and cpus ...

[nobo]

Why they don’t develop own Alberta Plan ?

[liminal]

I want a hackable AI machine for home use where I can experiment, procedurally generate content, etc. I want the Altair 8800 of machine learning and AI.

[shawnhagler]

I'm confused. Why do you keep re-uploading these?