Optimizing your SparkML pipelines using the latest features in Spark 2.3

preview_player
Показать описание
This talk will highlight the recent additions of vectorized UDFs and parallel cross-validation in Apache Spark 2.3. Vectorized UDFs not only enhance performance, but it also opens up more possibilities by using Pandas for input and output of the UDF. Parallel cross validation speeds up tuning ML models by exploiting your Spark cluster resources to the max. Bryan will discuss the details of Apache Arrow in Spark, how it is relevant to the rest of the big data ecosystem, and what is in store for the future. We will share performance results from using parallelism in cross-validation and some on-going work with optimizing ML pipelines.
This talk will also touch upon how we are leveraging this work to enhance the end to end Enterprise AI lifecycle in Open Source for developers and data scientists. Finally, we will highlight some of the other relevant projects at the Center for Open Source Data and AI Technologies and how you can contribute to the same.

Speakers
VIJAY BOMMIREDDIPALLI
Program Director: Center for Open Source Data and AI Technologies
IBM

BRYAN CUTLER
Software Engineer - Center for Open Source Data and AI Technologies
IBM
  1. DataWorks Summit
  2. Apache Arrow
  3. Apache Spark
  4. Jupyter
  5. Artificial Intelligence and Data Science
  6. DataWorks Summit San Jose
Рекомендации по теме
Optimizing your SparkML 0:43:38

Optimizing your SparkML pipelines using the latest features in Spark 2.3

Creating an Optimized 0:28:52

Creating an Optimized Data Pipeline for Data-Heavy Applications // Subsurface Summer 2020

Memory Optimization and 0:30:08

Memory Optimization and Reliable Metrics in ML Pipelines at Netflix

Extending Spark Machine 0:31:03

Extending Spark Machine Learning: Adding Your Own Algorithms and Tools

Productionizing Spark ML 0:43:47

Productionizing Spark ML pipelines with the portable format for analytics

MMLSpark: Lessons from 0:24:56

MMLSpark: Lessons from Building a SparkML Compatible Machine Learning Library -Miruna Oprescu

Optimizing, profiling and 0:47:55

Optimizing, profiling and deploying high performance Spark ML and TensorFlow AI models

The Killer Feature 0:24:30

The Killer Feature Store: Orchestrating Spark ML Pipelines and MLflow for Production

Best Practices for 0:41:20

Best Practices for Building and Deploying Data Pipelines in Apache Spark - Vicky Avison

Productionizing Spark ML 0:31:05

Productionizing Spark ML Pipelines with the Portable Format for Analytics (Nick Pentreath)

Automating Machine Learning 0:22:28

Automating Machine Learning Pipelines for Real Time Scoring (David Crespi)

Productionizing Spark ML 0:27:38

Productionizing Spark ML Pipelines with the Portable Format for Analytics

What Is Apache 0:02:39

What Is Apache Spark?

Tuning ML Models: 0:23:43

Tuning ML Models: Scaling, Workflows, and Architecture

Train Machine Learning 0:20:52

Train Machine Learning Model with SparkML (...and Python) | Hands-on tutorial

Extending Spark Machine 0:11:15

Extending Spark Machine Learning Pipelines Going beyond wordcount with Spark ML

Transformers, Estimators, and 0:04:12

Transformers, Estimators, and Pipelines (3/5)

Applying Multiple ML 0:26:37

Applying Multiple ML Pipelines to Heterogeneous Data Streams

Extending Spark Machine 0:32:04

Extending Spark Machine Learning Beyond Linear Regression by Holden Karau

Learn Apache Spark 0:10:47

Learn Apache Spark in 10 Minutes | Step by Step Guide

Performance Optimization of 0:32:50

Performance Optimization of Recommendation Training Pipeline at Netflix - Hua Jiang & DB Tsai

'Validating Big Data 0:39:56

'Validating Big Data Pipelines & ML (w Spark & Beam)' by Holden Karau

Building Custom ML 0:30:43

Building Custom ML PipelineStages for Feature Selection - Marc Kaminski

Extending Spark ML 0:30:15

Extending Spark ML to Deliver Fast, Scalable & Unified NLP