How to build and automate a python ETL pipeline with airflow on AWS EC2 | Data Engineering Project

In this data engineering project, we will learn how to build and automate an ETL process that can extract current weather data from open weather map API, transform the data and load the data into an S3 bucket using Apache Airflow. Apache Airflow is an open-source platform used for orchestrating and scheduling workflows of tasks and data pipelines. This project will be entirely carried out on AWS cloud platform.
We will cover the fundamental concepts of Apache Airflow such as DAG and Operators and I will show you how to install Apache airflow from scratch and schedule your ETL pipeline. I will also show you how to use sensor in your ETL pipeline.
As this is a hands-on project, I highly encourage you to first watch the video in its entirety without following along so that you can better understand the concepts and the workflows after which you should either try to replicate the example I showed without watching the video but consult the video when you are stuck or you could watch the video again the second time in its entirety while also following along this time.

Remember the best way to learn is by doing it yourself – Get your hands dirty!
If you have any questions or comments, ok to ask or leave comments in the comment section below.

*Books I recommend*

***************** Commands used in this video *****************
sudo apt update
sudo apt install python3-pip
sudo apt install python3.10-venv
python3 -m venv airflow_venv
sudo pip install pandas
sudo pip install s3fs
sudo pip install apache-airflow
airflow standalone
sudo apt install awscli
aws configure
aws sts get-session-token
***************** USEFUL LINKS *****************

Please don’t forget to LIKE, SHARE, COMMENT and SUBSCRIBE to our channel for more AWESOME videos.

DISCLAIMER: This video and description has affiliate links. This means when you buy through one of these links, we will receive a small commission and this is at no cost to you. This will help support us to continue making awesome and valuable contents for you.

[Zelinity]

This video has the signature of a master teacher. You introduced key concepts in a way that is simple to understand. Thank you for starting from level 1 without any assumptions of what we viewers/learners bring to the subject.

[vaibhavpawar]

Very detailed and basics tutorial with actual hands-on recorded. No PPT's simply a basic teaching which is very helpful for data engineer.

[hqn]

idk why you are not at least as hype as Zach Wilson. thank you very much for giving out high quality content for free!

[rex]

Thanks for taking the time to talk this video out!

[murilloalves]

This was just what I was looking for! Now it's time to apply it on my own projects. Keep the good work! Big thank you from Brazil!

[hellowillow_YT]

I've been looking for an ETL project videos that I can follow to learn basic data engineering stuff and finally I found your video! Thank you for this!

[mamadoulo]

Great work ! Thank for this project. I have already completed and i learn a lot.

[andresparra]

Thank you so much for this tutorial, was very helpful for an interview !!!

[hidjwhd]

i was initially puzzled or worried if i can grasp all but thanks for this video. this helps to dive in to code with airflow

[kerryw]

This is really good info! Thank you! One possible area to further advance this video is to upgrade the final task by loading data to an actual database (PostgreSQL for example).

[nnamdinwafor]

After watching this video, I knew I had to thank you for this truly awesome video. I have learnt more from this video than from many others out there. You are amazing.

[sachasmart]

Really good tutorial. Nicely done. Looking forward to part 2!

[mdobaidullahal-faruk]

Thank you very much for making the concepts so easy to understand👌

[seth_king_codes]

amazing man...would love more airflow/dags/python tutorials and also maybe how i can use with scraping data..cheers!

[narasa]

Excellent information, thank you so much for posting this video here

[mrgggvo]

congratulation, good job, part 2, dont forget

[gyungyoonpark]

this video is so clear and helpful. there are many airflow courses, but this video goes beyond and helps you "practice" airflow. hats off to the master and look forward to more awesome videos!!!

[collinspo]

This is a masterpiece. Thank you again and again !!!

[ndrimenan]

Thanks for this wonderful tuto. It’s time for me to practice now 🙏🏽

[pspointssara]

Excellent presentation. Even though I'm an experienced person still I need to learn a lot from your videos. This reminds me to watch more of your other videos in future. Good work and keep it up.