Tesla Dojo Supercomputer IN-DEPTH w/ James Douma

I'm joined by James Douma and guests Farzad Mesbahi and Nicolas Gibbs, as we watch Tesla's Dojo presentation and ask questions.

Jump to 2:09:41 to hear summary thoughts from guests
Jump to 2:36:02 to hear Part 2 in-depth look at Tesla Dojo hardware slides with James Douma and Dave Lee

Social

NOTE: Please don’t fall for scammers who might impersonate me or others in the comment section. I do NOT give out my number or ask people to contact me.

💁🏻‍♂️ Disclaimer: All content on this channel is for discussion and illustrative purposes only and should not be construed as professional financial advice or recommendation to buy or sell any securities. Should you need such advice, consult a licensed financial or tax advisor. All views expressed are personal opinion as of date of recording and are subject to change without responsibility to update views. No guarantee is given regarding the accuracy of information on this channel. Neither host or guests can be held responsible for any direct or incidental loss incurred by applying any of the information offered. Author is long TSLA and other stocks at time of original video publish date. When you buy through links in this video description, author may earn an affiliate commission.

Open an brokerage account at Interactive Brokers:

#Tesla #TSLA #Stocks

[patchin]

James Douma is an incredibly good teacher.

[ken]

I'm a hardware engineer at an AI start-up and, as usual, James' understanding and explanation is 100% spot-on.

[colinkelley]

My brain is hurting. I just can't take any more of this. but will come back and watch the second half tomorrow after sleeping for ten hours tonight. I am 78 and have a degree (1983) for programming IBM Mainframe computers: CICS, Cobol, Dos -- which, of course, is totally obsolete -- except I understood everything James was talking about! The underlying concepts and strategies were the same back then, just applied differently. James did an unbelievably fantastic job, and all your questions were so smart and on point. But think about this. If all the data streaming in from Tesla's million cars on he road were stored on paper punch cards, can you imagine how many punch cards it would take? It has only been 70 years. We have come a long way in just 70 years. Seventy years ago, the original computers cost $10.00 a bit -- not a byte, a bit --- TEN DOLLARS A FRIGGIN BIT! (and that was back when the dollar was worth 25 cents) Dave, thank you for doing this, and doing it so well. Now I understand why AI is advancing geometrically, and I now believe that FSD is inevitable -- and it is not too far away. In my opinion, Elon Musk will be traveling to Mars in about ten years -- because we are advancing in everything in geometric progression. The times we are living in makes the Industrial Revolution seem paleolithic.

[jweber]

James is so smart, such a good teacher, so nice and humble. He is the only person who can make me feel so stupid while enjoying it and leaving me asking for more!

[Thomas_Acharya]

Potential TOC (ready except some timestamps aren't perfect and maybe some more should be added):
0:00 Introduction (replays are always at 1.5x speed)
3:00 Dojo section starts
4:00 Dave interjects - Mission statement changing
6:45 Dojo section continues
6:57 Dave interjects - Month of compute and compiler
14:05 Dojo section continues
15:31 James interjects - Hardware constraints
28:32 Farzad's question about piecemealing moat
33:00? Better architecture now
34:05 Dojo section continues
35:17 Dojo section continues - Compiler technology
36:36 James interjects - Industry questions
39:00? Tiles, cabinets, chips
40:55 Nvidia equivalents
46:00 Dave's summary of equivalents discussion
47:57 How the training is done - Full vs partial
54:06 Dojo section continues
54:48 Dojo section continues - Material transition and CTE (Coefficient of Thermal Expansion)
55:30 James interjects - CTE and power/heat explanation
1:00:00 Plaid motor current clarification
1:01:06 Dojo section continues
1:02:00? James interjects - Clocks and voltage stepdowns
1:06:07 Dojo section continues
1:06:31 Dave interjects - James explains the resonance solution
1:08:00 Dojo section continues
1:08:55 James interjects - Cabinet cooling requirements and close integration
1:10:02 Dojo section continues - Interface Processors
1:11:30 James interjects - How to handle so many nodes (ingestion pathway)
1:14:00 How the tiles are oriented
1:14:42 Where the Operating System / compiler sits
1:16:00 Is the SRAM like L3 cache? It's the only memory
1:17:30 It's a deterministic synchronous system
1:18:38 What optimizations can they do? And ingestion pipeline info
1:24:02? How to organize jobs on the dies
1:25:16 Dojo section continues
1:26:27 Dojo section continues - Software
1:27:13 James interjects - Benchmarking the software
1:28:06 Dojo section continues
1:28:10 Linearly scaling jobs
1:28:56 Dojo section continues
1:29:19 James interjects - Coordination
1:23:35 Does the coordination have to happen, or is it due to software expertise?
1:35:36 Dojo section continues
1:35:48 How Dojo scales
1:36:45 Dojo section continues
1:37:02 James interjects Dojo hardware setup
1:38:07 Dojo section continues
1:38:25 James interjects - Compiler abstraction benefits
1:41:27 Dojo section continues
1:41:45 Dojo section continues - Stable Diffusion
1:41:56 James interjects - Stable Diffusion and how good the Dojo compiler is
1:43:14 GPT3 vs Stable Diffusion
1:44:45 Dojo section continues - Math summaries
1:45:18 James interjects - Batch norm and mathematical summaries
1:46:00 Dojo section continues
1:46:23 James interjects - How the batch norm is implemented and what would be ideal compared to what the compiler did
1:46:38 Dojo section continues
1:47:58 James interjects - Does this make sense?
1:48:04 Dojo section continues
1:48:07 James interjects - Dojo is really quick for batch norm 5 uS vs 150 uS
1:48:54 Dojo section continues
1:49:48 James interjects - What Tesla is running
1:50:10 Dojo section continues - Per die result
1:50:40 James interjects - Per die result analysis
1:52:25 Comparison using actual workloads / Partitions are different (assumes they optimized for both)
1:53:40 Large jobs being slower...
1:54:30 Tile production is a rate limiter
1:56:48 Number of GPUs
1:58:15 Dojo section continues - Overall results of how many can be replaced for same performance
1:58:39 James interjects - cost of GPU box and how many are replaced by Dojo
1:59:43 Power savings
2:17:06 How Tesla stages the data before the labeler?
2:18:24 Could the Voxels be shrunk?
2:23:22 Benefit of Dojo for increasing Tesla's moat
2:25:00? Tesla vertically integrates when others outsource
2:26:45? This enabled Tesla Bot
2:27:12 Compute is a raw material
2:29:15 Elon says factory is the product, is Dojo also like this?
2:31:00 Dojo makes sense
2:31:58 Dojo shows Tesla isn't just a car company
2:33:17 Surprise!
2:35:52 Dave and James first half video
2:38:00? DRAM deficiencies
2:41:10 SRAM vs DRAM
2:46:00? Fraction of top speed
2:56:58 Constraints SRAM only causes
3:01:00? Dojo can be split up
3:04:06 are TPUs doing something similar?
3:08:29 Dojo timeline
3:10:00? How many chips can they make?
3:17:45? Training tile
3:20:40 Integrated Circuit vs Chip
3:25:00? Voltage regulator module slide
3:26:10? CTE problem slide
3:27:50 Pushing Integration slide
3:28:45 1000 amps going into one D1 chip. It's more than the motor in your car (fuses blow). Not as much power though.
3:30:10 MEMS oscillator slide
3:34:10 Integrated solution slide
3:35:35 Cabinet slide
3:37:40 Dojo system slide
3:40:30? System tray slide
3:43:00 Dojo interface processor slide
3:48:00 How the network works
3:49:00? Another view of the Dojo interface
3:51:00? Dojo cabinet slide
3:51:08 ExaPOD slide
3:51:35 Software stack slide
3:56:58 Final comments

[nerdwatcher]

Professor Douma holding a class with a trio of YouTube personalities to make certain the message gets out through messengers who have an understanding of the topic. Show me another community with dedication like this! Thank you to everyone involved!

[NicholasShanks]

I need 35 hours in each day to keep up with Tesla news!

[spacegirl]

As usual James is outstanding at describing all of the technical aspects of DOJO! He’s so intelligent yet can take the technical and put it it layman’s terms so we can all understand.

[_mysilentblue]

This level of tech, information and interaction is absolutely priceless, Thanks Gents!!!

[shepherd]

Please keep these quality interviews coming! Thank you, Dave, James, Farzard, and Nicholas!

[PatternedPlanet]

Best word I heard from James in the original discussion was 'fabric'. That's a great idea that takes me back to my fiber days. It also succinctly describes that the size of the fabric is flexible.

[retiredbitjuggler]

Thank you! I have been so bummed that DOJO seems to have escaped most people. The bot was great, but DOJO really blew my mind. I guess most people just don’t get it, but there are a lot of people who must and few took notice. Why? This video did a great job and I hope it raises what people use to measure Tesla as a focused mega-tech company. Their current products are spectacular, and the efforts being pursued in manufacturing, energy, and IA are huge for the future of everyone. It is crazy how the average Joe is absolutely clueless. Thanks again. Can’t wait to finish the rest of the bonus portion of this video. I’ll just put on another pot of coffee. 😊

[babstude]

Holy crap fellas; that was an unbelievable amount of information to digest. Thank you very much for putting this together for us all !!! What a treat.

[shuriken]

Incredible knowledge that we are getting here. Many thanks to James for taking that amount of time to explain this to us, without asking for anything in return. Wish the Wall Street analysts would take the time and put in the effort to do something like this, as it is clear that they have no understanding of what was explained here.

[Dwizzle]

I saw 3:58:01 on the screen and was like DANG, I need to put this on my calendar, lol. Pt 1 was so good, can’t wait to dive into this one. 🙏🏾

[Snowcat]

This is amazing and precisely why I love the Tesla community! My deepest gratitude ❤

[chrisgerow]

James is marvelous. Hearing the video explanation - I get quickly lost. James fills in all the blanks.

[tomaspartl]

James Douma - thank you. It is like going to on-line university with you.
😉

[terryeasterday]

I just found your channel recently. I love how you truly want to understand what's going on at Tesla, and bring us along for the ride.😃👍

[cwmaguire]

Can listen to James Douma for hours. I'd love to hear a discussion with James, Karpathy and John Carmack.