I explain Fully Sharded Data Parallel (FSDP) and pipeline parallelism in 3D with Vision Pro

Build intuition about how scaling massive LLMs works. I cover two techniques for making LLM models train very fast, fully Sharded Data Parallel (FSDP) and pipeline parallelism in 3D with the Vision Pro. I'm excited to see how AR can help teach complex ideas easily.

Long time dream of mine to show conceptually, how I visualize these systems.

Chapters:

00:00 Introduction
01:02 Two machines each with 2 GPUs
01:37 Transformer models blocks
02:02 Forward pass
02:10 Backward pass
02:43 Fully Sharded Data Parallel introduction
02:51 Layer sharding
03:30 Weight concat
05:25 Memory upper bound
05:58 Why more GPUs speed up training
07:23 Shard across nodes (machines)
09:20 Sharding a block across nodes
10:14 Another way of seeing sharding
11:30 Understand interconnect bottleneck
12:00 Hybrid sharding
15:00 Pipeline parallelism
16:04 Forward pass in pipeline parallelism
16:10 Intuition around pipeline parallelism
16:50 Future directions on pipeline parallelism

[shinagawaintelligencecoltd]

i literally searched for a walk through like this for months. big big thanks.

[deependu__]

Great explanation. Loved the enthusiasm.

[Immurement]

This is really cool please do more of this even if it’s a bit buggy

[qiansun]

This is excellent! Very clear explanation with the help of vision pro! I do have one question, the fsdp seems pretty similar to model parallelism, how do they coordinate with each other when we enable both at the same time?

[juancolmenares]

Is sharding usually done on a layer level? Meaning, the distributed gpus will all have all the layers but only a piece of each layer? Or is it done like a pipeline, where one gpu feeds the output of a layer to another one?

[codewithyouml]

Hey really awesome video, but can you please translate this on iPad or a board? Thanks a lot