filmov
tv
Large Scale Data Loading and Data Preprocessing with Ray
Показать описание
(Wei Chen, NVIDIA)
Data loading is one of the most crucial steps in the DL pipeline. It needs to be designed and implemented in both a flexible and performant manner so that (1) it can be resued to support different DNN models, (2) it can match the speed of GPU compute, and (3) it can scale to multi-cores and even multi-nodes. However, achieving these design goals is not trivial, especially given that the most commonly used language in DL is python in which there is no good support for parallel programming.
In this talk, we will show that how we can use Ray to implement our data loading pipeline. Powered by the Ray actor, we are able to reuse most of our python modules and run our data loading pipeline in parallel without worrying about the overhead of managing it at scale. We will also talk about the experience and lessons we learned during our implementation and production depoyment.
Data loading is one of the most crucial steps in the DL pipeline. It needs to be designed and implemented in both a flexible and performant manner so that (1) it can be resued to support different DNN models, (2) it can match the speed of GPU compute, and (3) it can scale to multi-cores and even multi-nodes. However, achieving these design goals is not trivial, especially given that the most commonly used language in DL is python in which there is no good support for parallel programming.
In this talk, we will show that how we can use Ray to implement our data loading pipeline. Powered by the Ray actor, we are able to reuse most of our python modules and run our data loading pipeline in parallel without worrying about the overhead of managing it at scale. We will also talk about the experience and lessons we learned during our implementation and production depoyment.
0:22:29
Large Scale Data Loading and Data Preprocessing with Ray
0:40:39
Dr. Thomas Wollmann: Squirrel - Efficient Data Loading for Large-Scale Deep Learning
0:24:39
FAST '19 - DistCache: Provable Load Balancing for Large-Scale Storage Systems with Distributed....
0:12:25
NSDI '21 - MilliSort and MilliQuery: Large-Scale Data-Intensive Computing in Milliseconds
0:53:08
Building Large-scale Production Systems for Latency-sensitive Applications
0:05:25
What is Data Pipeline? | Why Is It So Popular?
0:08:42
7 Must-know Strategies to Scale Your Database
0:49:02
Data Movement at Very Large Scale | QCon SF 2013
0:02:12
Unleashing the Power of Big Data | A New Era of Analytics 🚀
0:11:46
Explore Fundamentals Of Data Analytics In Azure For Large Scale Data| K21Academy
0:24:44
Data modeling, the secret sauce of building & managing a large scale data warehouse | Citus Con ...
0:05:51
How-to | Conduct Large-Scale Data Warehousing with MaxCompute
0:53:05
Elasticsearch: Distributed System for Large Scale Data needs | Aravind Putrevu | GeekNight 63
0:52:11
Data Analysis Project | Large Scale Data Analysis | Switching from Pandas to FireDucks
1:35:19
Large-scale Data Process and ML Pipelines
0:05:22
Database vs Data Warehouse vs Data Lake | What is the Difference?
1:25:33
Resource-Efficient Redundancy for Large-Scale Data Processing and Storage Systems
0:10:04
Process HUGE Data Sets in Pandas
0:00:58
Data Pipeline Overview
0:00:57
How to scale a web application to a million users in 10 steps
0:18:26
| Microsoft Azure Data Fundamentals | Fundamentals of Large-Scale Data Warehousing |
0:09:24
System Design: Scale System From Zero To Million Users | #systemdesign
0:24:29
Large-scale data ingest on GCP (Google Cloud Next '17)
0:28:52
Unifying Large Scale Data Preprocessing and ML Pipelines with Ray Datasets | PyData Global 2021
Комментарии