Python Scrapy Tutorial - 22 - Web Scraping Amazon

We will be scraping the books department of amazon. More specifically the collection of books that were released in the last 30 days. Now if you are following along, you don't have to choose books. You can choose any department on amazon.

I already created the project 'AmazonTutorial' on pycharm and have installed scrapy. If you don't remember how to install scrapy you can always go back to my installing scrapy video.

Now before we run our spider, I just want you want to tell you that are program might not work. If you have scraped amazon before then it's not going to work but if this you first time then the above code will work. The reason for it not working is that amazon puts on restrictions when you are trying to scrape a lot of its data. So we are going to bypass those restriction by using something known as user agents. But before we get into that lets actually run our program

Next video - Bypass restrictions using User-Agents

#python

[beneditamontenegro]

I must congratulate you, on delivering a super user-friendly tutorial. I don't have much experience in programming and could complete it :)

[dnllln]

This is exactly what I was looking for and your teaching style is very thorough. Great job!

[larm_bee]

Awesome tutorial.I've been struggling for weeks and I managed to get it working in a day good job. Deserves my like and subscription

[NirmalSilwal]

this series is so so good and well explained, he never let me feel that it is boring and difficult. Just wow, great learning experience this entire scraping series. Thanks a ton.

[jackbird]

great tutorial, thanks. before it as a non-tech user, I am using to scrape amazon reviews e-scraper maybe it helps to somebody too.

[beauregs]

Very good series of tutorials. Congrats. The Amazon page has changed however. For the title, I decided to take it from the ALT text of the cover image (the cfMarker class isn't used anymore). For the price, they are now presented in various orders, not always the Hardcover price first (like in your example)... so I took the xpath approach to first find the span with hardcover in it, then went up 2 levels, before searching for the span with the amount. It can easily be adapted for Kindle and Audio book prices:

[shikharsrivastava]

Hello. What's the difference between .get() and .extract() ?

[cmanna]

This is amazing. Thank you so much for making this.

[rejowanahmed]

I am facing a problem. its only providing 25 entries. Not more than that. Please help

[arshiyasamreen]

I want 300+ refrigerator but I am getting only first 20 what I should do??

[bartsimpson]

@buildwithpython 10:17 is it some pycharm magic or you just stopped the video and paste by hand ?

[myvu-mvix]

when i am trying to get that developers.whatsmybrowser key I it is showing that my IP address is blocked
what should i do
???

[sankalparora]

what is the differnce between .extract() and .get()

[harshdwivedi]

@buildwithpython the order of the output is changed, first product_author is printing followed by name, link and price . And only price values are printed, all the other fields are returning empty arrays

[NA-cwpj]

Hi, thank you for this tutorial, make me understand a lil bit more all of this bout web scraping. But, i have a problem, constantly the terminal give me back "response is not defined", so i can´t go on to find all the values from the page. The thing is i try import all the libraries as you do, but still saying the same. Anyone, some answers ? Thank you all!

[ibekwekingsley]

I'm getting an error
AmazonScrapingItem does not support field: product_name

Any help

[GolightlyYeni]

what about pipelines and middlewares? In the editor it said that I have to define my item and the models for my spider middleware. But I couldn't see anything about that in this video

[ragafeb]

Awesome it's so helpful for me ..thanqq so much..

[symanhossenbux]

thank you for sharing, i belive that the selector gadget that you are using is extremely useful in the process of understanding the css paths. however, do you have any tips or resources for someone who is looking to also study the layout of css paths?

[axel]

Hi, very nice video!
Is it possible to webscrape some data and use that information to make a nice dashboard, tailored only to some data we are interested in?
Topic: real estate auctions