How To Scrape Woocommerce products with Python & requests-html

Lets scrape some more ecommerce sites, this time its getting the product data from a woocommce shop, using Python and requests-html only. No Beautifulsoup or Selenium needed. Using CSS selectors to scrape the links from the first page we loop through those and then grab each individual item page for the product info. Then using the build in CSV library for Python we create and export a new CSV file.

Support me:

Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases

[MindBlowerWTF]

This is my second python project and while I'm sure I don't understand most of stuff done here, I made it work with your help.

[mattmovesmountains]

Seeing the spreadsheet neatly arranged at the end of all this feels like I've done some sorcery.

[tubelessHuma]

More useful selectors. Thanks Dear John. 💖

[ferilukmansyah]

good tutorial ever !, thanks dear mr John

[LevelUpX]

Thank you! Can you let me know how to scrape product variations?

[rjasim]

I have some questions.
How can we extract images url?
Different variations, like size and color?
Also how to extract description?
Last as given code is not saving csv file please it would be great if you explain these things in next video.
Best Regards

[tanchunyeejoey]

Hi, any idea on how to scrape the add-to-cart variance?

[jagclub]

awesome work thanks, i just need to add url for the product image and the url for the product, hope you can help. thanks

[KhalilYasser]

Amazing as usual. Thank you very much. Did you post the code on GitHub?

[tulucartiom]

Hello @John Watson Rooney, please help me. This method works perfectly for the front page. How can I remove all the information if I have 10 product pages? Thank you

[Jigyasu_RP]

Amazing! But one thing I wanna ask, how do you scrape prices of variable products (in woocommerce as well as in case of amazon products)? Is there any way to do it? Please let me know.

[connorperzely]

If only it were this simple for Product Variants prices and attributes. I can't find a JSON object (for simple parsing) that contains the variant data and seems I'm forced into using Selenium. To get variant data I have to loop through, change the select inputs (to encapsulate all possible combos), and then record the data if there are products matching the select input filters. I noticed Shopify has a JSON object that you can easily parse but Woocommerce doesn't have it?

[murtazakalang]

I can't get any data from website where flex box is there.

[rogerhasemail]

nice sharing! how about AsyncHTMLSession? it would be much more efficient for multiple pages scraping

[husnainraza]

How can I scrap en.52wmb dot com and import key dot com please help.

[ProjectSkillsQMUL]

If I apply this method, how do you also scrape all the available pages? They are randomised URLs to prevent scraping but you can select the next page button by css or xpath. Sorry, I am extremely new to web scraping and python. Haven’t seen any examples of URLs when they become randomised

[o-henry]

Hmmm, was hoping there was some common woocommerce API that we could access. This method would still require me to write a scaper for each new site I wanna scrape

[achajackson]

ImportError: No module named requests_html
I have installed it with pip and pip3 but still does not work.

[georgegomes]

TypeError: get() missing 1 required positional argument: 'url'. Any idea why I am getting this?

[sseemm]

Why you didn't use r.html.render()
And is there a condition to use it or not?