Deep Dive into Stateful Stream Processing in Structured Streaming with Tathagata Das Continued

preview_player
Показать описание
Stateful processing is one of the most challenging aspects of distributed, fault-tolerant stream processing. The DataFrame APIs in Structured Streaming make it easy for the developer to express their stateful logic, either implicitly (streaming aggregations) or explicitly (mapGroupsWithState). However, there are a number of moving parts under the hood which makes all the magic possible. In this talk, I will dive deep into different stateful operations (streaming aggregations, deduplication and joins) and how they work under the hood in the Structured Streaming engine.

About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.

Connect with us:
  1. Databricks
  2. #SparkAISummit #apachespark
Рекомендации по теме
Deep Dive into 0:37:43

Deep Dive into Stateful Stream Processing in Structured Streaming - Tathagata Das

Deep Dive into 0:31:41

Deep Dive into Stateful Stream Processing in Structured Streaming with Tathagata Das (Databricks)

Deep Dive into 0:28:50

Deep Dive into Stateful Stream Processing in Structured Streaming with Tathagata Das Continued

A Deep Dive 0:48:40

A Deep Dive into Stateful Stream Processing in Structured Streaming 2018 Part 2 (Tathagata Das)

Google SWE teaches 0:15:08

Google SWE teaches systems design | EP40: Flink in 15 Minutes, Stateful Stream Processing!

Structured Streaming: Demystifying 0:37:23

Structured Streaming: Demystifying Arbitrary Stateful Operations

Introduction to Stateful 0:31:20

Introduction to Stateful Stream Processing with Apache Flink • Robert Metzger • GOTO 2019

Designing Structured Streaming 0:40:51

Designing Structured Streaming Pipelines—How to Architect Things Right - Tathagata Das Databricks

Tale of Stateful 0:30:59

Tale of Stateful Stream to Stream Processing

(25) Spark Streaming 0:19:16

(25) Spark Streaming : Stateless Vs Stateful operations Explained

Apache Spark 2.0: 0:29:25

Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das

#bbuzz: Stephan Ewen 0:37:23

#bbuzz: Stephan Ewen – From Stream Processor to Event-driven Database with Stateful Functions

DataXDay - The 0:48:29

DataXDay - The internals of stateful stream processing in Spark Structured Streaming

The Internals of 0:41:39

The Internals of Stateful Stream Processing in Spark Structured Streaming -Jacek Laskowski

Spark+AI Summit 2018 0:48:40

Spark+AI Summit 2018 - Stateful Stream Processing in Structured Streaming

Top 6 Most 0:04:21

Top 6 Most Popular API Architecture Styles

Stateful Stream Processing 0:05:12

Stateful Stream Processing with Flink SQL | Apache Flink 101

Stateless Vs Stateful 0:10:19

Stateless Vs Stateful transformations

How To Manage 0:09:37

How To Manage Stateful Streams with Apache Flink and Java

Sources, Sinks, and 0:35:53

Sources, Sinks, and Operators: A Performance Deep Dive

Keynote: Introducing Stateful 0:41:23

Keynote: Introducing Stateful Functions 2.0: Stream Processing meets Serverless Applications-S. Ewen

Keeping Redditors safe 0:30:34

Keeping Redditors safe in real-time with Flink Stateful Functions

FlinkNDB : Skyrocketing 0:23:54

FlinkNDB : Skyrocketing Stateful Capabilities of Apache Flink

How Netflix Ensures 0:49:32

How Netflix Ensures Highly-Reliable Online Stateful Systems