Apache Airflow for Data Science #4 - Migrate Airflow MetaData DB to Postgres and Enable Parallelism

preview_player
Показать описание
Apache Airflow doesn't run tasks in parallel by default - but there's an easy fix. Learn how to migrate MetaDB to Postgres and enable parallel execution.

00:00 Introduction
00:58 Modify Airflow configuration file
02:33 Initialize Airflow database and create the user
04:55 Restart Apache Airflow
06:06 Outro

FOLLOW BETTER DATA SCIENCE

FREE “LEARN DATA SCIENCE MASTERPLAN” EBOOK

GEAR I USE
  1. Better Data Science
  2. apache airflow
  3. airflow
  4. python
  5. data science
  6. programming
Рекомендации по теме
Learn Apache Airflow 0:12:38

Learn Apache Airflow in 10 Minutes | High-Paying Skills for Data Engineers

Axel Goblet: Scheduling 0:14:06

Axel Goblet: Scheduling machine learning pipelines using Apache Airflow | PyData Eindhoven 2019

Talk: Daniel Imberman 0:15:52

Talk: Daniel Imberman - Bridging Data Science and Data Infrastructure with Apache Airflow

Luther Hill: Use 0:38:03

Luther Hill: Use Apache-Airflow to Build Data Workflows | PyData Indy 2019

Building the Data 0:29:31

Building the Data Science Platform with Airflow at Near

Data Science Platform 0:44:24

Data Science Platform at PlayStation and Apache Airflow

Daniel Imberman & 0:33:47

Daniel Imberman & Gonzalo Diaz: Data Products, From 0 to data science pipeline | PyData Córdoba

Apache Airflow for 0:11:05

Apache Airflow for Data Science #7 - How to Work with REST APIs

Apache Airflow for 0:06:47

Apache Airflow for Data Science #4 - Migrate Airflow MetaData DB to Postgres and Enable Parallelism

Apache Airflow for 0:08:57

Apache Airflow for Data Science #10 - How to Work with Variables

Apache Airflow at 0:16:02

Apache Airflow at Wise

Apache Airflow Meetup: 0:25:49

Apache Airflow Meetup: Elegant Data Pipelining with Apache Airflow

Building (Better) Data 0:11:41

Building (Better) Data Pipelines with Apache Airflow

Data Engineering Course 7:00:25

Data Engineering Course | Apache Airflow Tutorial | Data Engineering Training | Intellipaat

Apache Airflow for 0:09:51

Apache Airflow for Data Science #1 - How to Install Airflow Locally in 10 Minutes

Airflow tutorial for 0:32:25

Airflow tutorial for beginners | Airflow tutorial python | Airflow tutorial 1

Simplifying the Creation 0:25:39

Simplifying the Creation of Data Science Pipelines with Airflow

Twitter Data Pipeline 0:40:21

Twitter Data Pipeline using Airflow for Beginners | Data Engineering Project

Apache Airflow in 0:00:53

Apache Airflow in under 60 seconds

Jean-Martin Archer & 0:24:37

Jean-Martin Archer & Michael Petro - Running Apache Airflow at Scale | PyData Global 2022

Data Science at 0:24:32

Data Science at Scale: From Notebook to Production with Airflow and Ray

Apache Airflow for 0:10:20

Apache Airflow for Data Science #3 - How to work With Databases (Postgres)

Apache Airflow for 0:00:57

Apache Airflow for Data Engineering | Apache Airflow | Data Engineering | Intellipaat #Shorts

Easy Airflow DAGs 0:01:18

Easy Airflow DAGs for ML and data science with Metaflow [no sound]

Комментарии
Автор

Thank you, much appreciated, piecing this togther from the documentation is not straight forward. Thank you.

Managed to connect my airflow cluster on WSL2 to postgres on the host and all is not smooth.

obiradaniel
Автор

Hello! Thanks for this amazing video. I am new to airflow, and i did the intial default setup in my local - using sqlite. But the scheduler goes down frequently and my task fails with following error: [2022-06-01, 02:30:24 IST] {local_task_job.py:154} INFO - Task exited with return code Negsignal.SIGKILL

[2022-06-01, 02:30:24 IST] {taskinstance.py:1267} INFO - Marking task as FAILED. dag_id=iris-pkg, task_id=split, execution_date=20220531T144529, start_date=20220531T152455, end_date=20220531T153024

[2022-06-01, 02:30:24 IST] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

While reading about this error, i found that i should switch to postgres and localexecutor. This video has helped me understand the theory - Can you please help me setup the airflow with postgres from scratch. I dont have postgres db installed on my local. So, would installing the airflow[postgres] serve the purpose or do i need to install postgres locally??

aditya