Two Dimensional Parallelism Using Distributed Tensors at PyTorch Conference 2022

Watch Meta AI's Wanchao Liang present his team's poster "Two Dimensional Parallelism Using Distributed Tensors" at PyTorch Conference 2022.

This talk will introduce 2-dimensional parallelism with PyTorch (Data Parallelism + Tensor Parallelism) using Distributed Tensor, a fundamental distributed primitive offered by PyTorch Distributed that empowers Tensor Parallelism. We have proven that using FSDP + Tensor Parallelism together could enable us to train large models like Transformer, and increase training performance. We offer end to end training techniques that enable you to train models in 2-D parallelism fashion, and checkpoint save/load models in a distributed manner.

#PyTorch #ArtificialIntelligence #MachineLearning