User Agent Switching - Python Web Scraping

Lets have a look at User Agents and web scraping with Python, to see how we can bypass some basic scraping protection. This video will show you what a user agent string looks like, how to find ones and how to send it along with your request, enabling you to convince the web server you are coming from a user controlled browser.

-------------------------------------
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases

[ddougher]

Hey John great tutorial. I just want to pass along to you and your audience that you should not scrape the WhatIsMyBrowser site. The warning along with an explanation for not doing so is located near the bottom of each user agent page.

[koala_]

Hey thanks a million mate!
Well spoken tutorial, love the calm tone too, 👍

[krispyren]

Very informative and helpful. Thank you John

[TheKrannyMaster]

Hey John, you are the man! You are really saving my ass at work for this web scraping project i am on!

[lanceradue]

Awesome, just what I was looking for to get thru to scrape headings on some websites!

[salehgoodarzian]

I can't thank you more for this video. It solved my problem....many many thanks

[purDvotion]

Great presentation and content showcase! Keep up the good work! 💪🏼

[itsmmdoha]

Just found you, loving your web scraping tutorials!

[nadaah]

Thank you so much for the video i was confused but this short video helped a lot

[rushilverma]

dude you just solved a thing that might take hours of mine searching online thanks a lot

[kikyou]

Your video's are amazing, thank you very much ❤️

[hiteshkathiriya]

Great Work... Appreciate Your help. Thanks.

[Jason]

Great video and this resolved my issue !

[juansebastiantobarcollazos]

Que chimba este video tienes un suscriptor nuevos

[theDaNi]

Hi John, just wanted to let you know that this video is featured twice in the Python Web Scraping playlist :)

[svetoslavkazandzhiev]

Great video John, I’m trying to avoid cloudflare and akamai protected websites. Would be great if you make videos about that.

[HasanAbiEdits]

I STARTED PYTHON YESTERDAY AND NOW I CAN

[matthiaswalther]

I tried to scrape a website with this method. Worked perfectly. However, after some tries and errors with the regard to the scraping process the error occurred again. the "503". Does that mean I got permanently blocked due to other reasons, i.e. my IP and my browsing behaviour? The website is called "Lieferando.de" and I tried to scrape the menu of a restaurant. I'mm fairly new to that topic. What can I do?

[AlphaMoury]

Excellent video! Is there any chance to change the IP in a dynamic way, so we don’t get blocked by the site via IP? Google works well for about 20 requests…

[napeters]

Hi John, what are the "other things" you can do besides changing user agents? I have tried different IPs, user agents, etc and yet somehow I still get 403 errors from a specific site, and this is before i even attempt to scrape it. I am just sending one request and i keep getting a 403 response. So curious as to how they know I am a bot and what else I can do. (and yes i have checked their robots file, they do allow bots)