Airflow, Spark, EMR - Building a Batch Data Pipeline by Emma Tang

preview_player
Показать описание
Robust and user friendly data pipelines are at the foundation of powerful analytics, machine learning, and is at the core of allowing companies scale with their data. In this talk, we will walk through how to get started building a batch processing data pipeline end to end using Airflow, Spark on EMR. Through real code and live examples we will explore one of the most popular OSS data pipeline stacks.
  1. DATA MINER
Рекомендации по теме
Building a Batch 0:41:11

Building a Batch Data Pipeline using Airflow, Spark, EMR & Snowflake

Airflow, Spark, EMR 0:38:15

Airflow, Spark, EMR - Building a Batch Data Pipeline by Emma Tang

How to submit 0:14:38

How to submit Spark jobs to EMR cluster from Airflow

Learn Apache Airflow 0:12:38

Learn Apache Airflow in 10 Minutes | High-Paying Skills for Data Engineers

Intro to Amazon 0:22:02

Intro to Amazon EMR - Big Data Tutorial using Spark

Loading data into 0:22:45

Loading data into S3 Iceberg Tables with AWS EMR and Apache Airflow

How to build 0:55:59

How to build and automate a ETL pipeline with AWS airflow | AWS End-To-End Data Engineering Project

Migrating Airflow-based Spark 0:39:00

Migrating Airflow-based Spark jobs to Kubernetes - the native way

Part 6 - 1:16:22

Part 6 - Create EMR cluster and Add steps tasks | Airflow Tutorial | Automate EMR Jobs with Airflow

Migrate Apache Oozie 0:24:17

Migrate Apache Oozie Workflows to Airflow and Run with Amazon EMR

Learn Apache Spark 0:10:47

Learn Apache Spark in 10 Minutes | Step by Step Guide

Airflow Tutorial | 2:37:28

Airflow Tutorial | Automate EMR ETL Jobs with Airflow | Airflow Project | Data Engineering Project

Running EMR jobs 0:10:31

Running EMR jobs with Airflow

Build a datapipeline 0:08:07

Build a datapipeline using EMR, Glue and Airflow in AWS

Running Spark jobs 0:25:12

Running Spark jobs on Amazon EMR Serverless

Run EMR on 0:07:58

Run EMR on EKS jobs on Apache Airflow

Build ELT Pipelines 1:55:04

Build ELT Pipelines using DBT and Spark on AWS EMR | Data Engineering

Building (Better) Data 0:11:41

Building (Better) Data Pipelines with Apache Airflow

AWS EMR Big 0:16:42

AWS EMR Big Data Processing with Spark and Hadoop | Python, PySpark, Step by Step Instructions

Migrating Airflow-based Apache 0:21:01

Migrating Airflow-based Apache Spark jobs to Kubernetes – the Native Way

Automating EMR Serverless 0:12:08

Automating EMR Serverless Workload |Creating|Submitting | Destroying EMR Cluster using Step Function

Amazon EMR on 0:12:20

Amazon EMR on EKS - Build Custom Images for Apache Spark on Kubernetes

Accelerate Amazon EMR 0:55:43

Accelerate Amazon EMR for Spark & More

Building a Data 0:40:14

Building a Data Lake on AWS with Apache Airflow

Комментарии
Автор

I appreciate how ambitious this presentation is. Spinning up all of these resources and using them in a live demo -- nicely done!

matpataki
Автор

Halfway in and I already admire you taking on the daunting task of spinning up things live. Also, the way you look at the screens up top is super cute.

AmanGarg
Автор

Emma Tang,
Can you please add the details of airflow, git hub resources in description?

saranyaelumalai
Автор

can you pls put GitHub link in description ?

dsinghr
Автор

@Emma, can you post the relevant github repos too? I'm trying to follow along this tutorial

musicisglobal
Автор

Awesome content, is there any GitHub repo for this content ?

PrakashReddyK
Автор

The presentation has a misleading title, it should have been titled - "let's play with Kubernetes and see what that is....and you also can run dockerized airflow on it. The end". So much time wasted.

dmitrysemenov