filmov
tv
Master Reading Spark Query Plans
Показать описание
Spark Performance Tuning
Dive deep into Apache Spark Query Plans to better understand how Apache Spark operates under the hood. We'll cover how Spark creates logical and physical plans, as well as the role of the Catalyst Optimizer in utilizing optimization techniques such as filter (predicate) pushdown and projection pushdown.
The video covers intermediate concepts of Apache Spark in-depth, detailed explanations on how to read the Spark UI, understand Apache Spark’s query plans through code snippets of various narrow and wide transformations like reading files, select, filter, join, group by, repartition, coalesce, hash partitioning, hashaggregate, round robin partitioning, range partitioning and sort-merge join. Understanding them is going to give you a grasp on reading Spark’s step-by-step thought process and help identify performance issues and possible optimizations.
Chapters:
00:00 Introduction
01:30 How Spark generates logical and physical plans?
04:46 Narrow transformations (filter, select, add or update columns) query plan explanation
09:02 Repartition query plan explanation
12:57 Coalesce query plan explanation
17:32 Joins query plan explanation
23:23 Group by count query plan explanation
27:04 Group by sum query plan explanation
28:05 Group by count distinct query plan explanation
33:59 Interesting observations on Spark’s query plans
36:56 When will predicate pushdown not work?
39:07 Thank you
#ApacheSpark #SparkPerformanceTuning #DataEngineering #SparkDAG #SparkOptimization
#dataengineering #interviewquestions #azuredataengineer
Dive deep into Apache Spark Query Plans to better understand how Apache Spark operates under the hood. We'll cover how Spark creates logical and physical plans, as well as the role of the Catalyst Optimizer in utilizing optimization techniques such as filter (predicate) pushdown and projection pushdown.
The video covers intermediate concepts of Apache Spark in-depth, detailed explanations on how to read the Spark UI, understand Apache Spark’s query plans through code snippets of various narrow and wide transformations like reading files, select, filter, join, group by, repartition, coalesce, hash partitioning, hashaggregate, round robin partitioning, range partitioning and sort-merge join. Understanding them is going to give you a grasp on reading Spark’s step-by-step thought process and help identify performance issues and possible optimizations.
Chapters:
00:00 Introduction
01:30 How Spark generates logical and physical plans?
04:46 Narrow transformations (filter, select, add or update columns) query plan explanation
09:02 Repartition query plan explanation
12:57 Coalesce query plan explanation
17:32 Joins query plan explanation
23:23 Group by count query plan explanation
27:04 Group by sum query plan explanation
28:05 Group by count distinct query plan explanation
33:59 Interesting observations on Spark’s query plans
36:56 When will predicate pushdown not work?
39:07 Thank you
#ApacheSpark #SparkPerformanceTuning #DataEngineering #SparkDAG #SparkOptimization
#dataengineering #interviewquestions #azuredataengineer
0:39:19
Master Reading Spark Query Plans
0:34:14
Master Reading Spark DAGs
0:33:12
Understanding Query Plans and Spark UIs - Xiao Li Databricks
0:03:58
04 Spark DataFrames & Execution Plans | Spark Logical and Physical Execution Planning | What are...
1:02:35
From Query Plan to Performance: Supercharging your Apache Spark Queries using the Spark UI SQL Tab
0:11:26
Spark Monitoring: Basics
0:44:13
Spark Execution Plans for Databricks
0:03:18
SQL Query Optimization - Tips for More Efficient Queries
0:08:24
Spark Logical & Physical Plan
0:04:47
95% reduction in Apache Spark processing time with correct usage of repartition() function
0:08:32
Spark Executor Core & Memory Explained
0:05:46
Spark Basics | Shuffling
0:10:47
Learn Apache Spark in 10 Minutes | Step by Step Guide
0:27:27
72. Databricks | Pyspark | Interview Question: Explain Plan
0:21:12
How to Read Spark DAGs | Rock the JVM
0:45:38
Adaptive Query Execution: Speeding Up Spark SQL at Runtime
0:30:19
Advancing Spark - Understanding the Spark UI
0:15:34
Catalyst Optimizer in Spark SQL| Logical Plan Vs Physical Plan
0:45:49
BeeScala 2016: Jacek Laskowski - Speak Spark SQL for better performance
0:41:35
Care and Feeding of Catalyst Optimizer
0:10:08
Apache Spark Mastery Roadmap - A Step-by-Step Guide to Unlock your Learning
0:07:37
Spark: Optimization Part2: Adaptive Query Framework| AQE| Spark 3.0 feature| Adaptive Query Plan
0:27:14
Master Databricks and Apache Spark Step by Step: Lesson 6 - Understanding Spark SQL (fixed sound)
0:25:48
Improving Interactive Querying Experience on Spark SQL
Комментарии