Web Scraping 101: A Million Dollar Project Idea

Web Scraping is one of the best projects you can work on with real, legitimate potential to make you a TON of MONEY ($$$)! It's the ability to collect real-time data about travel, eating, commerce, healthcare, real estate, you name it. Learn about this multi-billion dollar industry and how to tap into it!

🎬 Timestamps ⏱️
00:00 | Web Scraping
00:19 | The Potential
00:56 | The Idea
01:38 | The Process
02:12 | The Problem
02:47 | The Solution
03:36 | Million Dollar Project Demo
06:47 | Code Walkthrough
11:16 | BrightData Setup
12:14 | Next Steps & Ideas

◼️◼️◼️◼️◼️◼️◼️◼️◼️◼️◼️

◼️◼️◼️◼️◼️◼️◼️◼️◼️◼️◼️

⭐️ Tags ⭐️
- Tech with Tim
- Programming
- Web Scraping

⭐️ Hashtags ⭐️
#techwithtim #programming #coding

[TechWithTim]

Let me know any web scraping project ideas you have!

[flin]

So this is basically an elaborate commercial

[nyosdroid]

I did similar projects in the past (in php long time ago with curl and tidy, now in python with bs4 or selenium). To avoid bot detection, you can use the tor network, detect captcha request, and simply reconnect tor to get a different IP. Of course, you still need to put random sleeps here and there. Also, you need to be prepared to update or rewrite your code any time the website changes.
Nice video!

[TomGmaan]

I just turned 41 and awfully late to investing with barely any portfolio except my 401k, I have a decent amount of cash saved up and with inflation currently soaring AGAIN, I’m getting worried about retirement, my intention is to retire at 65 atleast, so how best do I maximize my savings of over $500k

[twentytwentyeight]

Just an anecdote, but when I first got into a data role I built a competitor price web scraper for my company. At the time it was supposed to be a show and tell for my boss in case python work came up in the future. It worked well enough for the business to utilize for beating the comp to vendors on price. We could keep tabs on them at a quick enough pace to adjust quotes and win. Didn't always work out but being able to say a python project netted three extra contracts is really cool

[voitkampff]

Great video! Thank you for talking about web scraping as an IT subset. What most people don't know is that it's extremely hard to build your career around it. I have been running my own web scraping software house for several years now and getting clients is more difficult than if you build apps, create websites, or do virtually anything else. People and companies simply don't have enough awareness what access to real time data can do for their businesses. Creating this awareness is actually the biggest part of my job. Writing code is just a formality afterwards.

[krvns]

Thumbs up if sponsored tools are not your content!

[blizzard]

this video is the reason removing the dislike button was a bad idea

[codesandtags]

I totally agree, I've done a couple of web scraping scripts for some companies, and to get some specific data, however there are a lot of challenges. But in general the rate limiting for APIs and changes in DOM, or catchaps are the most common ones.

[LeoTang]

I'm trying to build as many projects as I can right now to upgrade my skills and cv. This project idea is really helpful to my resume, I'll try to do it using fastapi and mysql (if possible I'll try to track stock price). Truly appreciated for sharing this project, sir!

[Coole]

20$ / GB is huge and you will hit that really fast lol

[nitindevatraj]

Need a whole series with videos like these

[x._ms]

youtube videos these days are becoming ads

[miyu]

A simple Chrome extension build would have made this so much easier for you and it's completely transparent for bot blocking.

[shitinsideyou]

Bright Data makes it easier to steal data from websites that try to protect themselves... Great job BrightData!

[yamani]

Just the cost of Bright Data makes this not applicable. I want to make 10, 000 requests a day at least. Anyone found a work around?

[pauljoseph]

I've seen people exploiting online job portal sites like Upwork. They hire freelancers from the Philippines to scrap websites rather than building a program, and of course they *CHEAP AF* like we're talking about $2-3 per 100+ websites.
Most of them charged $2-3 per hour which is absolutely disgusting.

[alainportant]

Ok so it's an ad for a paid service.

Fuck that, if I wanted a paid solution I would have looked for a paid solution already, not a video tutorial.

[SamuXZamorionVII]

He is right, I work for tech consulting company. One of the client projects I work on is web scraping project that collects doctor disciplinary action data using python and they pay a lot for us to do that for them.

[workind]

Good idea. If u r in 2013. That's plenty of services who are provided same possibilities