Optimizing Apache Spark SQL at LinkedIn

preview_player
Показать описание
Presenter: Fangshi Li

Presented at the Bay Area Apache Spark Meetup hosted at LinkedIn in August 2019.

Abstract: Improving the Spark SQL usability and computing efficiency is one of the missions for Linkedin’s Spark team. In this talk, we will present the Spark SQL ecosystem and roadmaps at Linkedin, and introduce the highlighted projects we are working on, such as:
* Improving Dataset performance with automated column pruning
* Bringing an efficient 2d join algorithm to Spark SQL
* Fixing join skewness with adaptive execution
* Enhancing the cost-optimizer with a history-based learning approach

Bio: Fangshi Li is a software engineer at Linkedin. He has been working on Spark core infrastructure, user libraries, AI solutions, and Spark SQL engine optimizations. He was one of the original developers of Dr. Elephant, the performance tuning tool for Hadoop/Spark.
  1. LinkedIn Engineering
Рекомендации по теме
Optimizing Apache Spark 0:20:20

Optimizing Apache Spark SQL at LinkedIn

Optimizing Apache Spark 0:29:33

Optimizing Apache Spark SQL Joins: Spark Summit East talk by Vida Ha

Apache Spark Core—Deep 1:30:18

Apache Spark Core—Deep Dive—Proper Optimization Daniel Tomes Databricks

Exploring Spark SQL 1:06:43

Exploring Spark SQL Optimizations Part-1 | Spark SQL | Apache Spark | Optimizations

95% reduction in 0:04:47

95% reduction in Apache Spark processing time with correct usage of repartition() function

Spark SQL: Another 0:16:19

Spark SQL: Another 16x Faster After Tungsten: Spark Summit East talk by Brad Carlile

Adaptive Query Execution: 0:45:38

Adaptive Query Execution: Speeding Up Spark SQL at Runtime

How We Optimize 0:21:25

How We Optimize Spark SQL Jobs With parallel and sync IO

Apache Spark for 0:38:52

Apache Spark for Machine Learning on Large Data Sets • Juliet Hougland • YOW! 2017

From Query Plan 1:02:35

From Query Plan to Performance: Supercharging your Apache Spark Queries using the Spark UI SQL Tab

Optimizing Apache Spark 0:18:10

Optimizing Apache Spark UDFs

Apache Spark Joins 0:11:21

Apache Spark Joins for Optimization | PySpark Tutorial

SQL Performance Improvements 0:25:47

SQL Performance Improvements at a Glance in Apache Spark 3.0

Secret To Optimizing 0:05:57

Secret To Optimizing SQL Queries - Understand The SQL Execution Order

Spark performance optimization 0:20:20

Spark performance optimization Part1 | How to do performance optimization in spark

Deep Dive into 0:34:15

Deep Dive into Query Execution in Spark SQL 2 3 with Jacek Laskowski

Understanding Query Plans 0:33:12

Understanding Query Plans and Spark UIs - Xiao Li Databricks

Optimize read from 0:34:53

Optimize read from Relational Databases using Spark

Fine Tuning and 0:25:19

Fine Tuning and Enhancing Performance of Apache Spark Jobs

Understanding the Working 0:00:50

Understanding the Working of Apache Spark's Catalyst Optimizer in Improving the Query Performan...

optimization in spark 0:13:03

optimization in spark

Spark Basics | 0:05:13

Spark Basics | Partitions

Apache Spark Core 1:32:03

Apache Spark Core – Practical Optimization Daniel Tomes (Databricks)

Cost Based Optimizer 0:35:20

Cost Based Optimizer in Apache Spark 2 2 - Ron Hu & Sameer Agarwal

Комментарии
Автор

The 2d partitioned join looks really promising. Especially for star schema kind of table where a fact can be joined with multiple dimension. Is there any open sourced sample that we can use?

JoHeN