Advancing Data Mesh: Building A Stream Processing EcoSystem of Reusable Processors and Datasets

Data Mesh is Netflix's next-generation stream processing platform for enabling real-time data movement at scale supporting multiple user personas. Instead of point-to-point data movement which usually leads to duplicate data and diverged processing patterns, Data Mesh strives to move data in a bus architecture to reduce engineering and resource overhead and make both data and processing patterns reusable. Data Mesh is an open collaborative platform where data sources and processor patterns are contributed by platform and partner teams. Data Mesh abstracts Kafka as a Transport layer and Flink applications as Processors. It also manages schema evolution across the entire system seamlessly for both compatible and incompatible changes. In this talk, we'll explain Data Mesh's user personas & architecture, dive into reusable components & schema evolution, and share our experiences with operating the platform.

0:00 Intro
1:06 Why Data Mesh
3:20 Challenges
19:04 Current Status, Future Plans