Scrape Amazon NEW METHOD with Python 2020

Whilst working ona new personal project i noticed that scraping amazon with requests and bs4 no longer worked, so I am sharing a new method of how to get prices and titles from any amazon product page.

-------------------------------------
-------------------------------------
-------------------------------------
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------
Sound like me:

-------------------------------------
Video like me:

-------------------------------------
PC Stuff:

[mattmovesmountains]

Subscribed from this video. It was paced well and did a great job at isolating the main functionality here without burying it in complex implementations.

[TheKrannyMaster]

Hey bro - thanks for this! Ran into the issue using bs4 - thanks for showing me another option!

[randomstuff.]

I was stuck with the old one. Thanks for new one

[quentinfitzgerald]

RuntimeError: Cannot use HTMLSession within an existing event loop. Use AsyncHTMLSession instead. Trying to figure out AsynchgHTMLSession is a bit off scope for me at the minute as Im just trying to learn the basics of using requests and bs4. Just wondering why you didn't get that error John as you ran it multiple times. I'm running my script in a jupyter notebook would that be a problem?

[ghaithmoe]

The problem with "requests_html" that it's too slow, So I'm using "lxml" instead of using "html.parser", it works fine.
Thanks for your hard work John.

[TheKturner]

Hey John I watched your video where you scraped the wkisky site with scrapy and I was excited to get started scraping amazon with scrapy but this method you dont seem to use scrapy? instead opting for a bsoup integration. Why dont you utilize scrapy with Amazon?

[rickynguyen]

Hi, John, thanks for your video. It was great and I've learned much from your scraping video. I have a website that i want to scrape but the url has longtitude and latitude within it. I have used many of your technique to scrape but it was unsuccessful. Is there differerent approach to scrape data from url with coordination ? Thanks

[MaloneMatty]

Have you noticed overnight that Amazon have changed the pagination elements?

class_="s-pagination-item s-pagination-next s-pagination-button s-pagination-separator" (the new class) doesn't render in Soup, but looks like it still appears as <li class='a-last'> element.

[ferilukmansyah]

I just scraped amazon, I want to ask how to deploy to a production server, heroku for example

[ugurdev]

I can not get html-requests to ever run unfortunately, throws an exception when it is trying to download chromium. Tried to insert the file manually inside the folder it wants to create, but still ignores it and wants to download. Google wasn't much help, I also wonder if I could just use geckodriver.

[yunfeiericzhao]

Hi John, do you save the code somewhere so I can simply copy and run it and see how it works?

[arcosd]

What semantics are you writing? Is Request_HTML similar to Python semantics?

[im]

Nice.Just Subbed.How do you learn new things.Books?

[mhdshahul]

hi sir, thanks for this amazing video !
i am trying to do same for search result when we have lots of tv with model and price, (the page before this one). But it shows only the example i chose to have the xpath...
do you have a recomendation ?

[alexgreat]

how long can your code run before it returned 503 ?

[nakjkro]

Thank you Bro, You help my assignment. Subscribed.

[mubeenkhan]

Hi John,
Can you please tell me how can i set this up in google sheets and then get the data from title colum with many entries.?

[Tooske.]

Thanks for the tutorial helped me quite alot,
still i got a little issue with the price, for e.g 'price': 100, 99\xa0€' how can i fix it ?

[catesconsultinggroupllc]

@John Watson Rooney Do you have any insight as to why I wouldn't be getting any results in the terminal at all? I'm not receiving an error or results. I've tried commenting out r.html.render(sleep=1) in case it was timing out.

[Nachodark]

thank you so much for the vid!
I have a question...
How can i get the full price (price+tax+shipping)? when i try to run the code, simply copying the xpath of it i get an error (i believe it is because i am trying to get the value form a table, as i am opening the "DETAILS" to see the full price.

English is not my first language, so sorry if i didnt explain myself really well!