Basic Data Engineering Project - End-To-End From Web Scraping to Tableau

Building your first data engineering project can be daunting. There are a lot of tools you're supposed to be able to use that range from Python and SQL to the Cloud.

So let's make this project simple and scrape some data, parse it using SQL and loading it into Tableau.

Get A $25 Credit For Bright Data Using The Link Below If You Book a Demo

Raw Data

Timeline
0:00 Intro
3:53 Scraping Data
10:00 Reviewing Data
12:11 Working On Snowflake
29:12 Working On Tableau

If you enjoyed this video, check out some of my other top videos.

Top Courses To Become A Data Engineer In 2022

What Is The Modern Data Stack - Intro To Data Infrastructure Part 1

If you would like to learn more about data engineering, then check out Googles GCP certificate

If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.

Or check out my blog

And if you want to support the channel, then you can become a paid member of my newsletter

Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio

_____________________________________________________________
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.

*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

[KrisPatel]

Thanks Ben! I used something like this to fiqure out Opendoor listings across the Atlanta region and ran it against what they bought at and how much decline they are seeing.

It was fun excersize.

Kind of cool how much free information is out there as long as your willing to scrap the data and make sense of it.

[splashoui]

I liked the snowflake part of this video.,

[thetruthsayer]

Thanks for making this video. I’ve been contemplating switching to data engineering. This video just reminded me how boring data is to me😂😂😂😂

[shunsuikyoraku]

Hello Ben, I just wanted to say that I would love to watch your videos with a better sound quality 😁

[moeheinaung]

SQL parts in snowflake are really confusing and not well explained. Are the scripts available somewhere?

[TheWizardTower]

Great content! Just a suggestion, it might be helpful to slow down and enunciate your words a bit more for better understanding. Keep up the good work!

[macuriels]

Hi! Recent subscriber here👋 I'm very much a novice on the subject - are the tools you use free? If not, are the free alternatives you can recommend?

[Ty-exkb]

Thank you for the vid!! Not sure if this question is obvious or not as i’m VERY much a novice but when you’re working on snowflake (~17:14) where you using SQL queries or coding in a different language? Just not sure what you used to parse everything

[Andrew.Skinner]

Thanks Ben. Any resources on swapping out a Python selenium script downloading a csv file for the scraping portion in this vid?

[dongyan]

Is snowflake free to use to build this project for my portfolio? I only see a trial

[arvindsuriakanth]

For the Tableau part, can we use Tableau public or should we use Tableau Desktop

[parthvisoni]

Do we need to pay for using the api from bright data?

[BISimplifier]

Good content.
As a sort of native speaker you are speaking too fast and audience would find it difficult to pace with.

[Dev__hire]

This vidoe isn't beginner-friendly!

[FRUXT]

Hard to follow what you saying when you are not english speaker because you don't articulate...

[nnn]

For projects, do you think it is good to code components like the scraper from scratch like python or is it better to use existing tools? I feel like showing you can code, and know the very granular components of scraping data is great, but showing you use tools that people have heard of is also another argument. The reason being, I wrote a python script that scrapes data using regex, and runs hourly and transforms and loads into postgres. Do you think it would be beneficial to use an existing tool like you used from a hiring manager's perspective?