How to Rotate Proxies with Python

In this video I go through how to implement rotating proxies using requests with Python. We look at scraping some free proxies and writing a script to see if they work. Although free proxies aren't any use for actual web scraping projects the principles are the same.

If you are new, welcome! I am John, a self taught Python (and Go, kinda..) developer working in the web and data space. I specialize in data extraction and JSON web API's both server and client. If you like programming and web content as much as I do, you can subscribe for weekly content.

:: Links ::

:: Disclaimer ::
Some/all of the links above are affiliate links. By clicking on these links I receive a small commission should you chose to purchase any services or items.

[TipsTat]

Great video.
I think 3 things are REALLY worthy of note because the answer to the failures isn't necessarily that the proxies were blocked:
- Sometimes the proxies weren't able to be connected to in the first place
- Sometimes the Server can't handle so many back to back requests

AND
Some proxies are set up by malicious actors to gain unauthorized access to connecting hosts.

[irfanshaikh]

9:28 this part changed my programming outlook to drastic extent.
The wonders you can pull using the threaded approach is just sublime.
Thanks John once again.
It seems im running out of gratitudes to give you the due credit.

[TheFredro]

residential proxies were mostly fine after switching to Privoxity

[marxlenin]

Ive been binging your content because of an amazon scraper I'm working on.

Can't help but giggle at the consistent struggle you have in typing "requests"

All in good fun! Keep up the great content!

[chizzlemo]

Nice to see a video where I understand everything first time! Thanks

[TECHWIZARD]

Checked more than 10 videos, all videos are awesome
Thanks Bro

[hypebeastuchiha]

Every time I have a python scraping problem I search your name
Great channel mate 👍

[RS-Amsterdam]

Thanks to the man who has all the answers to my questions.
Man, you have the well of wisdom on behalve of scraping/python, JSON and what matters in that field.
Although I am a pro IT guy for many many years (PM, Consultant, Architect and Advisor) this field of expertise is rather unexplored for me,
but following your video's made it crystal clear to me.
Thanks again for sharing

[abhinavshah]

Super Insightful, John!
Thank you for sharing this one 🙌

[bagia]

when i choose a proxy and print the status code, it returns 200, but when I want to show the json, it returns exception. why?

[Pre.cadaver]

Beautiful content as always! I tried scraping the site in this video using bs4 as it's the only framework I know as of now. I hope you make a video on scraping this site as you said in the video.

[MorganDavis]

Thanks! Do you know any services with rotating proxies that rotate every 60 seconds and were you can choose mix geo or a specific country? I have mine from proxy-stоrе but this is my first service, want to have cheap alternatives and to find out other options

[irfankalam]

Thanks for this video and looking forward to your future video about working proxies which are useful for web scraping

[martpagente]

Thankyou John! It's getting more interesting everytime you upload videos. By the way, can you start using Request+BeautifulSoup+Splash sooner in the future, especially in setting up? I'm have a couple of questions for you next Q&A series, I'm excited!

[lightblue]

Thanks chief, informative and easy to understand tutorial

[XELTRA]

i try your code, the problem is 2:59, the response is 200, when i use print(r.json) there is error so go to except, but without json, proxylist show working, please tell me why print(r) and print(r.json) are different result

[ghaithmoe]

This trick saved my time. Thank you 🙌🏾

[celerystalk]

New skill learned. Thank you as always!

[lucifervns]

Hello even request.get response value 200 for a url and it looks like proxy is working
but when we load a this website etc then it's always show can't access, load timeout, rendering timeout..etc. So do we have any way to check those proxy could work as normal? Thank you so much

[hippolytesucche]

Nice video but how can you deal with auth proxies?