Web Scraping with BeautifulSoup and CSS Selectors

This week I am revisiting a web scraper that I wrote almost a year ago, and improving and cleaning it up. Using requests, beautifulsoup and some user defined functions we deal with pagination and export products to a csv file using pandas. Itertools helps to flatten out a list of lists and is a handy solution to know.

Learning Python with Web Scraping is a good fun and practical way to improve your coding abilities. It was interesting for me to see how much I have improved in the last year going over and rewriting a scraping script.

Support Me:

-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------

[martin-xqte]

Your videos are brilliant. Just one more to add to your list. Thank you

[tnssajivasudevan]

One more great example of knowledge sharing. Great Sir. Pls keep going
🙏

[__wouks__]

Always something interesting to learn and I have learned quite a lot from your videos. Just a note that with results.append(productinfo) you could have changed it to results.extend(productinfo) and then you wouldn't need to do the itertools.chain operation.

[wangdanny]

Hey John. Every bit of John per day. I just ecounter some problems but I re-visited the first one on whisky. It is solved. Thank you very much. I will try my best to watch every vid you have and keep up your weekly updates. Many thanks.

[andriikorniienko]

this video desires a million views, thank you sir

[mylordlucifer]

Great Job, from your learning i am become strong Thanks,

[infocloudbiz]

Thank you for this video .Its very clear explanation. Expecting detail video for web scarping jobs search.

[vladimirantonov]

Thanks a lot! Very interesting and informative!! :))

[samman]

I agree CSS selectors are better to use. Not only are they easier, they're also more robust.

[mushinart]

Great content, lived the way you used irtloos

[prod.kashkari]

Also do a video on scheduling web scrapers with Apache airflow

[donaldandmijung]

thank you this tutorial! do you have a video parsing data out of a table using css selectors?

[saloni.kataria]

Do you have any video on how to scrape from multiple websites and can we scrape from an entire website rather than a particular category? Thanks

[imrushB]

Hey, great video! Could you explain how to make datacenter proxies in a server? Using a provider

[mrsfunny]

Hi how to export scraping result to markdown format (hugo) ? Thanks

[apk]

Any chance of a video comparing web scrapers and parsers for speed and reliability, threading and non-threading?

Think it would be a good one coming from a master with recommendations, pros and cons.

I'm looking for a reliable and speedy system for js websites but one page at a time after building one based on your videos using bs and selenium. Not sure if it's fast enough compared to others.

[prod.kashkari]

U could just use the extend() function which appends the elements from the list to another without adding the whole entire list.

[violence]

How can I select different css selectors from different webs?

[huzaifaameer]

hi there! can you use Regular expressions in scraping content like name, price, sku etc...

[robertbalazsi]

Great videos, John! I'd be curious about your take on CSS Selectors vs XPath. That is, which one would you recommend over the other in certain scenarios.