Apache Airflow for Data Science #2 - How to Write Your First Airflow DAG (Data Pipeline)

preview_player
Показать описание
Apache Airflow is a common tool used by Data Engineers. Learn how to write your first data pipeline (DAG) in 10 minutes.

00:00 Introduction
00:37 Initial setup
01:21 Airflow DAG boilerplate code
04:07 Task #1 - Get current datetime
05:46 Task #2 - Process current datetime
09:50 Task #3 - Save processed datetime
14:45 How to connect the Airflow DAG
16:26 How to run the Airflow DAG
17:36 Outro

FOLLOW BETTER DATA SCIENCE

FREE “LEARN DATA SCIENCE MASTERPLAN” EBOOK

GEAR I USE
  1. Better Data Science
  2. airflow
  3. apache airflow
  4. python
  5. data engineering
  6. data science
Рекомендации по теме
Building the Data 0:29:31

Building the Data Science Platform with Airflow at Near

Learn Apache Airflow 0:12:38

Learn Apache Airflow in 10 Minutes | High-Paying Skills for Data Engineers

Axel Goblet: Scheduling 0:14:06

Axel Goblet: Scheduling machine learning pipelines using Apache Airflow | PyData Eindhoven 2019

Data Science Platform 0:44:24

Data Science Platform at PlayStation and Apache Airflow

Talk: Daniel Imberman 0:15:52

Talk: Daniel Imberman - Bridging Data Science and Data Infrastructure with Apache Airflow

Luther Hill: Use 0:38:03

Luther Hill: Use Apache-Airflow to Build Data Workflows | PyData Indy 2019

Elegant data pipelining 0:35:38

Elegant data pipelining with Apache Airflow - Bolke de Bruin

Building (Better) Data 0:11:41

Building (Better) Data Pipelines with Apache Airflow

Apache Airflow for 0:00:57

Apache Airflow for Data Engineering | Apache Airflow | Data Engineering | Intellipaat #Shorts

Machine Learning with 0:23:54

Machine Learning with Apache Airflow

Apache Airflow at 0:16:02

Apache Airflow at Wise

Easy Airflow DAGs 0:01:18

Easy Airflow DAGs for ML and data science with Metaflow [no sound]

Apache Airflow in 0:00:53

Apache Airflow in under 60 seconds

Apache Airflow Meetup: 0:25:49

Apache Airflow Meetup: Elegant Data Pipelining with Apache Airflow

Building Efficient Data 0:19:56

Building Efficient Data Pipelines with Airflow | Step-by-Step Tutorial | Apache Airflow

Simplifying the Creation 0:25:39

Simplifying the Creation of Data Science Pipelines with Airflow

An Introduction to 0:55:11

An Introduction to Apache Airflow | DataHour by Kapil Madan

Apache Airflow for 0:09:51

Apache Airflow for Data Science #1 - How to Install Airflow Locally in 10 Minutes

Airflow tutorial for 0:32:25

Airflow tutorial for beginners | Airflow tutorial python | Airflow tutorial 1

Data Science at 0:24:32

Data Science at Scale: From Notebook to Production with Airflow and Ray

Apache Airflow for 0:08:57

Apache Airflow for Data Science #10 - How to Work with Variables

Apache Airflow for 0:09:37

Apache Airflow for Data Science #6 - How to Communicate Between Tasks with Airflow XComs

Apache Airflow One 0:51:35

Apache Airflow One Shot- Building End To End ETL Pipeline Using AirFlow And Astro

Defining data pipelines 0:35:34

Defining data pipelines workflows using Apache Airflow - Juan Riaza

Комментарии
Автор

I appreciate your efforts for this video. It deserves more than likes and comments. Great Job Mate

RajeshSamson
Автор

Thank you, Dario. I have only recently started learning Apache Airflow.

paleface_brother
Автор

how come after i follow the steps for part 1 and this one, i have airflow imports could not resolved? i have installed it in part one.

damiencheung
Автор

the process_datetime keeps giving an error

damolaolayinka-osho
Автор

ModuleNotFoundError: No module named 'pandas'
how to solve this error bro

karthickraja
Автор

File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 171, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 189, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/postgres_db_dag.py", line 24, in process_iris_data
raise Exception('No data.')

Could you help me sir? where i went wrong?

rohmankpaii
Автор

Hi @Better Data Science ..
I wrot the same script as you but ti.xcom_pull return NoneType instead of the datetime .. do you have a solution please ? thanks

salimmzoughi