Model vs Data Parallelism in Machine Learning

preview_player
Показать описание

  1. Mark Saroufim
Рекомендации по теме
Model vs Data 0:09:32

Model vs Data Parallelism in Machine Learning

Unit 9.3 | 0:02:24

Unit 9.3 | Deep Dive into Data Parallelism | Part 1 | Understanding Data Parallelism

Model Parallelism vs 0:06:59

Model Parallelism vs Data Parallelism vs Tensor Parallelism | #deeplearning #llms

Concurrency Vs Parallelism! 0:04:13

Concurrency Vs Parallelism!

Part 2: What 0:03:16

Part 2: What is Distributed Data Parallel (DDP)

Parallel Computing Explained 0:03:38

Parallel Computing Explained In 3 Minutes

A friendly introduction 0:24:19

A friendly introduction to distributed training (ML Tech Talks)

Task Parallelism - 0:00:54

Task Parallelism - Intro to Parallel Programming

EfficientML.ai Lecture 19 1:09:57

EfficientML.ai Lecture 19 - Distributed Training Part 1 (Zoom Recording) (MIT 6.5940, Fall 2024)

Task Parallelism 0:06:13

Task Parallelism

Data Parallelism Using 0:27:11

Data Parallelism Using PyTorch DDP | NVAITC Webinar

Training on multiple 0:05:35

Training on multiple GPUs and multi-node training with PyTorch DistributedDataParallel

Is it concurrent 0:03:48

Is it concurrent or parallel?

How Fully Sharded 0:32:31

How Fully Sharded Data Parallel (FSDP) works?

Unit 9.2 | 0:06:56

Unit 9.2 | Multi-GPU Training Strategies | Part 2 | Choosing a Multi-GPU Strategy

ChatGPT vs Thousands 0:13:26

ChatGPT vs Thousands of GPUs! || How ML Models Train at Scale!

SysML 19: Jia 0:26:11

SysML 19: Jia Zhihao, Beyond Data and Model Parallelism for Deep Neural Networks

3. Data Parallelism 0:29:20

3. Data Parallelism

MALT: distributed data-parallelism 0:26:04

MALT: distributed data-parallelism for existing ML applications

Training LLMs at 0:55:59

Training LLMs at Scale - Deepak Narayanan | Stanford MLSys #83

Distributed Training with 1:12:53

Distributed Training with PyTorch: complete tutorial with cloud infrastructure and code

Generalized Pipeline Parallelism 0:20:50

Generalized Pipeline Parallelism for DNN Training

Data parallelism difference 0:00:18

Data parallelism difference between data parallelism vs task parallelism parallel algorithm CSE 6th

CMU 10714 Project 0:01:30

CMU 10714 Project - Distributed Training : Model Parallel

Комментарии
Автор

That was great, i specially liked it that you showed some simple code snippets.

moeinhasani
Автор

The first code sample I show is an example of model parallelism
The second code sample I show is an example of data parallelism

I want to discuss pipelining in another video

marksaroufim
Автор

Are the graphs correct? It looks like when the data or model size is increased the time converges to some value. It would mean that we can double data or model size without any increase in time.

adhithadias
Автор

Thanks, Mark! Can you tell what's being exchanged across nodes in either of these cases? My understanding is that in data parallelism the weights and biases are shared and eventually their averages are shared across all the nodes. Not sure what's being shared between nodes in model parallelism.

VivekSharma-tmkd