I built a distributed scraping system, but was it worth it?

➡ WORK WITH ME

➡ COMMUNITY

➡ PROXIES

➡ HOSTING

If you are new, welcome. I'm John, a self taught Python developer working in the web and data space. I specialize in data extraction and automation. If you like programming and web content as much as I do, you can subscribe for weekly content.

⚠ DISCLAIMER
Some/all of the links above are affiliate links. By clicking on these links I receive a small commission should you chose to purchase any services or items.

This video was sponsored by ProxyScrape.

[graczew]

🔥🔥🔥 as always. Thanks for your content.

[Aidas_Li]

I need to do 70k daily…. 😢 still learning. Managed 40k with no block.

[onealoneal]

Still there when I receive notification 🎉. Thank you for your amazing content

[DeviceDuo-slrb]

Hello, I entered your channel and found many videos. Where do you advise me to start to learn web scraping?

[dave]

you run the risk of ddossing the host. i ran a scraping project with 1M+ a day and that was on a single server, single threaded, and with exponential backoff if it hit any errors. no need for any proxies and the extra latency and errors that come with those.

[alexdin]

thanks John, please make more tutorial about proxy services reviews videos so we can compare between them

[muhammadfahim]

same thing can be achieved with multiprocessing.