Python Scrapy Tutorial - 24 - Bypass Restrictions using Proxies

In this last video we bypassed the scraping restrictions by using user-agents and in this video we will be learning how to bypass them by using something known as proxies.

Before we go into proxies, you need to understand what is an IP address. An IP address is basically an address of your computer. You can find your own IP address by going to google and typing in 'What is my IP'.

Whenever you connect to a website you are automatically telling them your IP address. A website like amazon can recognize your IP address and ban you if you try to scrape a lot of it's data. But what if used a another IP address instead of our own. And even better we can use a lot of IP addresses that our not our own, and put them in rotation. So we every-time we send a request to amazon. It's going to be with a different IP address.

When you use an IP address that is not your own. Then that other IP address is known as a proxy. If we look up the definition of proxy on google it says 'the authority to represent someone else'. So basically we are hiding our address and using someone elses.

Next video - Scraping multiple page of amazon

#python

[leslievanelsie]

I will say it once again, this is the best SCRAPY tutorial on YouTube
Thank you for all the great stuff you've taught me
Conceivable explanations

[didovecigor]

If you look at logs it wasn't actually scraped "properly", it was scraped with host IP, exactly same issue that happens to me. Nonetheless great video series and I'd appreciate continuation with more in depth material. I think there is a big lack of content in this field so think about it twice. Thanks for the videos and good luck.

[davyroger]

This is great! i know this is old but a video dealing with browser fingerprinting, cookies and other methods of bot detection would really complete the tutorial. Thanks again man, learned a lot

Btw if any one is getting a Response.text error try uninstalling than reinstalling scrapy-proxy-pool==0.1.7
as the versions after that validate the response.text

[SandwichMitGurke]

this was way easier than i expected. this tutorial series has helped me so much, thank you

[khurramjaved]

Thanks man i was trying to scrap a site for i don't know ? may 2 days, after building scraper for it ! then i spent other 3 days, trying to get their data, but site was continuously sending me to user login, data extraction rate was like 10 % out of all tries, watching your 23rd video just make my day, Bam ! now i am fkg scraping that site :D .. Love For You <3

[Imu]

Thanks for sharing your knowledge with us. You are good teacher. Can you please add a video in this list about how can we connect to SQL server and add the the scraping data to database. It would be much appreciated if you do this.

[safi]

Hi Attreya, thanks for the amazing video. despite trying both bypassing technic on your playlist, I was unable to to access the pages I was trying to crawl. do you have other suggestions or technic? Thanks. keep it up the good works.

[hrishabhgupta]

I have a question like I am trying to scrape the data using scrapy, On website there is data but in my scrape response i am getting \xa0 only. Any Idea how to fix this issue??

[atultanna]

Can we bypass google with proxies? In short to use it with Scrapebox
Hope the code was available to make it more visible

[raghavendrasinghchouhan]

You are amazing bro, , god bless you.. you reach new hights, ,, you always cover everything, whichever i need

[sfrgvh]

Hi,

Thanks for the video!

Is there any restriction on crawled number?

scrapy creates a csv that ends with 200 rows but should be 3700 rows

[NishithSavla]

Great content sir but this proxy method isn't working somehow. It is raising an AttributeError: Response content isn't text. Can you help me with this?

[mayurbarbhaya]

hi there!
thanks for wonderful tutorial!

can i run those user-agent middleware from python single script using scrapy?

[ilyasstaybi]

hey i have a problem and can't find where the issue is, whenever i run a scrapy projet it returns "crawled 0pages.." any help would be usefull

[Funnyanimalstalkk]

bro will you make one video on how to solve recaptcha

[twittertrendings]

How to scrape the data when website provide access within the country not for other countries. For example: I want to scrape an e-commerce website data but being a foreigner I cannot scrape the data of the website which doesn't show me the html code.

[donrak]

hey ! thank you very much, this is exactly what i needed:)


i just have 1 problem: the scrapping only works like every 3. time. even after i did use user agent and proxy_pool, any idea?

[shaikhanuman]

hi Sir if we add proxy configuaration in settings.py file we comment the chrome user agents or not ?

[amriteshmadhur]

@buidwithpython could you please add how to use both proxies and user agent together that would be great .May be just in description just snippet of settings.py file. Content is good . Thanks in advance

[puneethc]

can we scrape as many images as possible? Using this technique, I have observed that we can scrape only a few