Getting Started with Airflow for Beginners

Getting Started with Airflow for Beginners.

At the end of this video, you will be able to:
✅ Identify the different ways of installing and running Airflow in local
✅ Create a local dev environment with Airflow following best practices
✅ Create data pipelines using Variables, XComs, and the Taskflow API
✅ Share data between tasks
✅ Trigger and monitor data pipelines
and more.

If you're entirely new to Apache Airflow, take a look at those videos first:

00:00 Introduction
00:28 Install and setup Airflow in local
00:38 Pip install
00:57 The docker-compose file
01:51 The Astro CLI
03:46 Airflowctl
04:17 Reminder of the basics
04:30 The DAG you will create
05:00 Create the DAG
07:08 Create the first task
08:55 Create the second task
09:24 Airflow variables
10:55 Create the last task
11:15 Define task dependencies
12:35 The final DAG
13:05 Run it!
13:33 Use the UI to monitor and manage your DAGs

With that, you have all the keys to getting started with Airflow!

[kirillfedorov]

Today I watched the entire playlist without a break, you have a talent for explaining! Thank you. Also, perhaps the last two videos would be better placed at the beginning of the playlist.

[Incomingsign]

i have setup airflow with docker using astro cli. how to make code suggestion work in vscode?

[sophialawal]

Thank you the video is really help full

[Robino_del_Bosquet]

Many thanks for the video ; great tutorial !

I have a question regarding Airflow though ; I keep seeing DAG file where task functions are defined directly into the DAG file.
What if I want to import functions from python scripts from other Repositories ? How can I call them in my DAG file ?

I plan to use Airflow to run many scripts I have deployed on various Git repositories.
I don't want to recreate those already existing functions into my DAG file but just call those scripts with their functions in my Airflow repository.
Any idea how can I achieve that ?

Many thanks and keep up making great videos ! ;)

[kazaki]

Using tempfile module from python, can I create a temporay directory associated with a DAG RUN that will be automatically deleted at the end of a DAG RUN regardless if the DAG RUN was successful or failure or otherwise?

[sebastian]

Great video Marc! But why doesn't Visual Studio Code recognize the Airflow library even though I'm running my containers correctly? I've noticed that you're experiencing the same issue because the 'airflow.decorators' library is underlined in yellow. Is there a way to fix this?

[baloney_sandwich]

why the video resolution looks messed up lol