filmov
tv
Improving Apache Spark Application Processing Time by Configurations, Code Optimizations, etc.
Показать описание
In this session, we'll go over several use-cases and describe the process of improving our spark structured streaming application micro-batch time from ~55 to ~30 seconds in several steps.
Our app is processing ~ 700 MB/s of compressed data, it has very strict KPIs, and it is using several technologies and frameworks such as: Spark 3.1, Kafka, Azure Blob Storage, AKS and Java 11.
We'll share our work and experience in those fields, and go over a few tips to create better Spark structured streaming applications.
The main areas that will be discussed are: Spark Configuration changes, code optimizations and the implementation of the Spark custom data source.
Connect with us:
Our app is processing ~ 700 MB/s of compressed data, it has very strict KPIs, and it is using several technologies and frameworks such as: Spark 3.1, Kafka, Azure Blob Storage, AKS and Java 11.
We'll share our work and experience in those fields, and go over a few tips to create better Spark structured streaming applications.
The main areas that will be discussed are: Spark Configuration changes, code optimizations and the implementation of the Spark custom data source.
Connect with us:
0:24:46
0:04:47
![[Tech Talk] Enhancing](https://i.ytimg.com/vi/A3C-39VwjiA/hqdefault.jpg)
0:14:36
0:27:59
0:29:16
0:31:28
0:29:29
0:30:37
0:20:20
0:29:27
0:40:57
0:33:10
0:05:33
0:26:34
0:32:41
0:22:35
0:43:42
0:27:58
0:33:20
0:00:59
0:20:30
0:00:50
0:36:13
0:21:59