Build your first pipeline DAG | Apache airflow for beginners

#apacheairflow #airflowforbeginners #datapipeline #etl #etlpipeline #airflow2

Welcome to series Apache airflow for beginners. In this video, you'll learn how to build your first pipeline DAG in Apache Airflow. As a Data Engineer, you will be tasked to write a data aggregation and cleaning pipeline. We are going to build an ETL pipeline as part of Airflow DAG with a real-world example of Hotel Reservation Records.

---------------------------------------------------
Get complete code from here;

Watch next video
Airflow Macros

Watch the previous video here:
Run Airflow locally via Docker

--------------------------------------------------------
Timeline

00:00 intro
00:13 definition
00:40 DAG building blocks
01:39 Example - ingestion pipeline overview
03:05 Lets Code DAG
07:53 Lets deploy DAG
09:29 Examine DAG output
10:25 Ingest more data
11:17 Outro

--------------------------------------------------------

Learn Something New Below:

👍 If this video was helpful to you, please don't forget to thumbs-up,
⌨️ leave a comment and
🔗 share with your friends.

⭐ Support our work by hitting that subscribe button.
🔔 hit bell icon to stay alert on more upcoming interesting stuff.

Stay tuned !

[MrMegabeat]

My best 10mins investment in the morning! 🎉

[demohub]

This video has definitely given a better understanding of Airflow and now have some ideas on how to use it more effectively for projects.

[muhammedalbayati]

Thanks a lot. Very good tutorials. Thumbs-up and subscribe

[najmuddin]

Thanks for the tutorial! However, could explain what the difference is between running this workflow as an airflow DAG and simply running the program and calling both functions sequentially?

[parikshitchavan]

Hello Thanks for such a great tutorial everting you made smooth like butter, just one question whenever we made new DAG we will have to add (docker-compose-CeleryExecutor, docker-compose-LocalExecutor, and Config for that particular DAG )??

[downeytang]

a quick question, if you have two different date format from two csv files, and after performing concat, is there a way to unify the date format, for example '2021-01-13' and '01-20-2022'

[victoriwuoha]

@MaxcoTec Please do you have any resource on how I can read data from an API and perform some similar processing & finally write to a destination SQL server. I'll be grateful if you could advise.

[nghianguyen]

Thank for very good videos. Can you help me to give some instructions about an example data pipeline in Mongodb?

[FedericoLov]

good video but it seems that the actual transformations are done in pandas while airflow only provides a layer of logging and task scheduling

[ammadkhan]

how can I access airflow container when I am hosting to another server and add more dags?

[hsk]

hey brother in my ui dag is not possible

[riyasingh]

my task are getting failed in airflow UI can u tell why it is happening, so I copied all your code properly

[muhammedalbayati]

Please how can save these CSV data to MS Sql server database?

[orick]

You should delete "beginners" title from the headline...

[TheVickramsharma]

Hi @MaxcoTec, i tried running this example and am getting this error: FileNotFoundError: [Errno 2] No such file or directory: '/opt/airflow/raw_data/2022-12-13/5/booking.csv', could you please help me in this