Robust Data Processing Pipeline with Elixir and Flow - Laszlo Bacsi - ElixirConf EU 2018

preview_player
Показать описание
It doesn't matter if you have "big data" or "small data" if you need to import and process it in near realtime you want to have a system that is robust and maintainable. This is where the failure tolerance and scalability of Erlang/OTP, the expressiveness of Elixir, and the flexibility of Flow and GenStage are all great assets. This is a story of how we built a data pipeline to move and process billions of rows from MySQL and CSV files into Redshift and what we learned along the way.
  1. Erlang Solutions
  2. Elixir
Рекомендации по теме
Robust Data Processing 0:41:13

Robust Data Processing Pipeline with Elixir and Flow - Laszlo Bacsi - ElixirConf EU 2018

Building a Robust 0:42:34

Building a Robust Data Pipeline with the 'dag Stack': dbt, Airflow, and Great Expectations

What is Data 0:10:34

What is Data Pipeline | How to design Data Pipeline ? - ETL vs Data pipeline (2024)

Building Robust and 0:27:03

Building Robust and Scalable Data Pipelines with Kafka

What is Data 0:05:25

What is Data Pipeline? | Why Is It So Popular?

Robust Foundation for 0:38:18

Robust Foundation for Data Pipelines at Scale - Lessons from Netflix

Building a robust 0:20:31

Building a robust data pipeline with dbt, Airflow, and Great Expectations

AWS re:Invent 2015 0:34:23

AWS re:Invent 2015 | (CMP310) Robust Data Processing Pipelines Using Containers & Spot Instances

Data Preprocessing in 0:22:01

Data Preprocessing in AI/ML - Part 2: A Guide for AI Enthusiasts (All about AI) - Machine Learning

Building Robust Data 1:59:29

Building Robust Data Pipelines for Modern Data Engineering | End to End Data Engineering Project

Paul Brebner - 0:24:13

Paul Brebner - Building and Scaling a Robust Zero-Code Data Pipeline With Open Source Technologies

Best Practices for 0:27:18

Best Practices for Building Robust Data Platform with Apache Spark and Delta

Beam Summit 2021-Leveraging 0:37:55

Beam Summit 2021-Leveraging Beam's Batch-Mode for Robust Recoveries and Late-Data Processing

Operationalizing Big Data 0:23:31

Operationalizing Big Data Pipelines At Scale

How To DESIGN 0:31:03

How To DESIGN YOUR First DATA PIPELINE ??🔥 15 Minutes BASIC STEPS

Jiaqi Liu - 0:20:16

Jiaqi Liu - Building a Data Pipeline with Testing in Mind - PyCon 2018

Designing the Next 0:27:01

Designing the Next Generation of Data Pipelines at Zillow with Apache Spark

Data to Deployment 0:19:05

Data to Deployment - Crafting a Robust Machine Learning Pipeline

[Tech Talk] Enhancing 0:14:36

[Tech Talk] Enhancing Apache Spark for robust data processing

What is a 0:00:20

What is a Data Pipeline Engineer?

Reduce Update Latency 0:41:19

Reduce Update Latency and Build a Robust Pipeline Architecture for Ingesting at Scale

Stream vs Batch 0:09:02

Stream vs Batch processing explained with examples

Data Pipeline Lifecycle: 0:31:13

Data Pipeline Lifecycle: SQL Everywhere

Let’s Make Your 0:15:12

Let’s Make Your Data Pipeline Robust With Great Expectations!｜Keisuke Nishitani｜PyCon Taiwan 2021...

Комментарии
Автор

Good talk. Note that the introductory part is over at 18:40. You may want to just skip over it.

JXSMS
Автор

Thanks for the talk. I have a bit of a side-topic question: how are you streaming objects to and from S3 buckets? From what I can tell, ex_aws_s3 doesn't support streaming, but it is possible to stream files using HTTP calls if they are made available on a URL i.e. either publicly or using a pre-signed URL. Is there any other way I'm missing?

yawaramin
Автор

1:42 - Mic problem is taken care of and talk starts.

zesky