Twitter Data Pipeline using Airflow for Beginners | Data Engineering Project

USE CODE: COMBO50 for a 50% discount

This is End-To-End Data Engineering Project using Airflow and Python. In this project, we will extract data using Twitter API, use python to transform data, deploy the code on Airflow/EC2 and save the final result on Amazon S3

Other resources:

Timestamps
0:00 Project introduction
0:56 Section 1 (Prerequisite)
2:21 Section 2 (Understand basic concept)
7:55 Section 3 (Project execution)

🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟

3 Books You Should Read

Tech I use every day

✨ Tags ✨
data engineering project, complete data engineering project for beginners, learn data engineering, data engineering roadmap

✨ Hashtags ✨
#airflow #dataengineering #project

[DarshilParmar]

EDIT:
Elon Musk ruined the API part, you can use some other APIs


or you can use a static dataset from Kaggle and then use Airflow to process it

FAQ:

2. You need to request for V2 access: You will get an error but just by reading the doc you should be able to access Elevated Access


Worked very hard for this project🤞
Don’t forget to hit the like button and if you want to support my work you can join channel membership, it's only 59rs per month so that I can keep these content for free

[sumantaghosh]

this is what is called content and actual do something for community ..rather than just doing faltu things...thank u so much Darshil ❤️

[hirenparkar]

Attention ❗❗❗❗❗❗❗
Anyone trying to do this project please check this imp note
Twitter has made changes to its API access levels, and some v1.1 endpoints are no longer freely available.
so that's why I wasn't able to do this project.

Thankyou Darshil bro for this wonderful project, even if I wasn't able to execute this project. I was able to understand about the project.
Thanks again for helping all of us in this Big Data Engineering journey

[ANKITKUMAR-nvur]

In data engineering community you are the GEM. Thanks from the bottom of my heart Darshil. Keep growing keep inspiring.

[AshisRaj]

Mr. Darshil you have saved a lot of time for us by this video. This video has more content than others talkative videos of other youtubers. Thanku so much

[ThankGod]

I came here seeing your LinkedIn post with comments, Being a fresher I was struggling for a project. This is Absolutely what I was looking for. Just Awsome. Thank you so much for your time and efforts.

[swapneil]

Twitter stopped free access to their APIs :|

[owennowakowski]

This was a very helpful video for me. I have spent the last year learning web development, and I am expanding my skills to include ETL. This video gave me a great overview of the process.

[kanduladivya]

This video is worth more than paid content on many websites.Thanks for such content.

[venkatah]

My sincere thanks to your efforts for contributing these kind of real time scenario based use case examples. 🙏

[faramarzzrn]

one of the best tutorials I've ever seen on Youtube, a real-world example that was really interesting

[sujeetnair]

i m elder than u.. but u r an inspiration bro... i m new to data engineering..

[akashsoni]

Those who are stuck in the api part can look for some other apis and perform ETL on it too and make your own project.
Darshil helped us to understand the concepts therefore its not mandatory to use only Twitter api.
Thanks a lot Darshil :)

[aaravkumarsingh]

Thanku darshil for making this project, please make this type of project on regular basis so that it will help us to enhance our skill

[dele_data]

I find it worthy to join a youtube channel membership for the first time. I hope with smart and harwork from myself and your help, I am able to land a Data Engineering job soon.

[nomadicengineer]

Please keep on bringing content like this it will be really helpful for freshers who want to understand, gain knowledge and shape their career in Data engineering🙏.

[abdulrazakshaik]

Really Appreciate your efforts to the Data Community you are doing a great job. Thank you ❤️

[chandrakantsingh]

Thanks for creating this project, with the help of this i schedule my data extraction task, currently i am using solcast API to fetch weather data .

[tassamoo]

Thanks sir, I learn a lot following this project guide. I understand more about how to use tweepy, AWS services and airflow by following this tutorial. I hope you always have a nice day.

[laminann]

This video really help me to build my first data pipeline with a different API, it is really helpful.
Thank you Darshil.