filmov
tv
Alpa: Automating Inter- and Intra- Operator Parallelism for Distributed Deep Learning
Показать описание
SAMPL Talk 2022/03/03
Title: Alpa: Automating Inter- and Intra- Operator Parallelism for Distributed Deep Learning
Presenter: Lianmin Zheng (UC Berkeley)
Abstract: Alpa automates model-parallel training of large deep learning (DL) models by generating execution plans that unify data, operator, and pipeline parallelism. Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations, which does not suffice to scale out complex DL models on distributed compute devices. Alpa distributes the training of large DL models by viewing parallelisms as two hierarchical levels: inter-operator and intra-operator parallelisms. Based on it, Alpa constructs a new hierarchical space for massive model-parallel execution plans. Alpa designs a number of compilation passes to automatically derive the optimal parallel execution plan in each independent parallelism level and implements an efficient runtime to orchestrate the two-level parallel execution on distributed compute devices. Our evaluation shows Alpa generates parallelization plans that match or outperform hand-tuned model-parallel training systems even on models they are designed for. Unlike specialized systems, Alpa also generalizes to models with heterogeneous architectures and models without manually-designed plans.
Bio: Lianmin is a third-year Ph.D. student in the EECS department at UC Berkeley, advised by Ion Stoica and Joseph E. Gonzalez. His research interests lie in the intersection of machine learning and programming systems, especially domain-specific compilers for accelerated and scalable deep learning.
-
SAMPL is an interdisciplinary machine learning research group exploring problems spanning multiple layers of the system stack including deep learning frameworks, specialized hardware for training and inference, new intermediate representations, differentiable programming, and various applications. We are part of the Paul G. Allen School of Computer Science & Engineering at the University of Washington. Our group is a collaboration between researchers from Sampa, Syslab, MODE, and PLSE.
Title: Alpa: Automating Inter- and Intra- Operator Parallelism for Distributed Deep Learning
Presenter: Lianmin Zheng (UC Berkeley)
Abstract: Alpa automates model-parallel training of large deep learning (DL) models by generating execution plans that unify data, operator, and pipeline parallelism. Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations, which does not suffice to scale out complex DL models on distributed compute devices. Alpa distributes the training of large DL models by viewing parallelisms as two hierarchical levels: inter-operator and intra-operator parallelisms. Based on it, Alpa constructs a new hierarchical space for massive model-parallel execution plans. Alpa designs a number of compilation passes to automatically derive the optimal parallel execution plan in each independent parallelism level and implements an efficient runtime to orchestrate the two-level parallel execution on distributed compute devices. Our evaluation shows Alpa generates parallelization plans that match or outperform hand-tuned model-parallel training systems even on models they are designed for. Unlike specialized systems, Alpa also generalizes to models with heterogeneous architectures and models without manually-designed plans.
Bio: Lianmin is a third-year Ph.D. student in the EECS department at UC Berkeley, advised by Ion Stoica and Joseph E. Gonzalez. His research interests lie in the intersection of machine learning and programming systems, especially domain-specific compilers for accelerated and scalable deep learning.
-
SAMPL is an interdisciplinary machine learning research group exploring problems spanning multiple layers of the system stack including deep learning frameworks, specialized hardware for training and inference, new intermediate representations, differentiable programming, and various applications. We are part of the Paul G. Allen School of Computer Science & Engineering at the University of Washington. Our group is a collaboration between researchers from Sampa, Syslab, MODE, and PLSE.
0:17:39
OSDI '22 - Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
0:56:39
Alpa: Automating Inter- and Intra- Operator Parallelism for Distributed Deep Learning
0:25:00
Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
1:00:07
RAS: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning - Gustavo Leite
0:55:07
Alpa: Automated Model-Parallel Deep Learning - Zhuohan Li | Stanford MLSys #59
0:31:08
Alpa - Simple large model training and inference on Ray
0:24:40
Alpa A Compiler for Distributed Deep Learning - TVMCon2023
0:15:15
OSDI '23 - AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
0:00:41
dependence automatic parallelization
0:12:33
USENIX Security '21 - Using Amnesia to Detect Credential Database Breaches
0:20:47
SysML 19: Zhihao Jia, Optimizing DNN Computation with Relaxed Graph Substitutions
0:30:48
Building Brains - Parallel training strategies of large-scale deep learning neural networks
0:03:09
Hybrid Parallelism
0:26:11
SysML 19: Jia Zhihao, Beyond Data and Model Parallelism for Deep Neural Networks
0:06:55
Ray Community and the Ray Ecosystem
1:11:40
Review of Quality Control Considerations for Resting-state fMRI: Dr. Jean Chen
0:10:46
Parallel Training of Deep Networks with Local Updates
0:29:17
FHPNC 2021 - Parallelism-preserving automatic differentiation for second-order array languages
0:28:59
Evaluating large language models with Ray in hybrid cloud
0:06:23
GCA:122 Distributed Deep Learning of ResNet50 and VGG16 with Pipeline Parallelism
0:24:30
Paper #107. Varuna: Scalable, Low-cost Training of Massive Deep Learning Models
1:00:31
High-Performance Communication Strategies in Parallel and Distributed Deep Learning
0:14:11
USENIX ATC '21 - Fine-tuning giant neural networks on commodity hardware with automatic pipelin...
0:52:25
Parallelism and Acceleration for Large Language Models with Bryan Catanzaro - #507
Комментарии