Learn Web Scraping With Python: Full Project - HTML, Save to CSV, Pagination

This video includes a complete how to of web scraping with Python for beginners. I go through scraping an HTML website, talking about why I have done what I've done and the reasons behind my decisions. It includes, getting data from the server, parsing HTML to get the product information, using functions to not repeat code and make it more organized, dealing with pagination, some small error handling and exporting the final data to a CSV file. Follow along and learn how to web scrape / data extraction with Python.

Support Me:

-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------

[irfanshaikh]

John at 36:26 - if you guys are interested in watching
People like me - well off course. I aint got anything better to do. Ill be here again

Thanks John

[CB]

Hey John, only been getting into scrapy over the past few weeks and have gone back and watched a lot of your videos (maybe 20). V high quality content and your channel has a huge amount of rewatchability.

My next steps will be in connecting my scrapy spiders to a db. Keep up the great work :)

[mushinart]

always go back to basics ... mastering the basics means mastering the advanced stuff ... amazing work as usual bro ...love to chat/video call you n say hi, god bless

[SaMi-seqs]

Hey John! I think I almost watched your all videos.Whenever I get a problem I found solution for that somewhere in your videos.Thank you for uploading such precious contents.

[mk]

I like the the fact that when you do not like smiling, you don't fake it just for more subscriber. A lovely earnest dude.

[sandilemfazi]

You are a life saver, i have binge watched all your youtube videos, i learned so much, Thank you so much

[stelvio]

Great John...i wait for update of this project ;) thank you so much!

[shanicedoesdata]

This tutorial was so good! Used it to collect some data for a peronal project of mine!

[wangdanny]

Hey John, thanks again. I watched another episode where you mentioned combination of selenium page_source and beautifulsoup. So I skipped htmlsession since it didn't work on my side. BUT it perfectly worked out in the end. THANK YOU!!! Love your voice.

[CodePhiles]

great demonstration for full code for scraping a website with pagination, Thank you John, ... for me, I had been evaluated lately for such a task to be hired as a scraper as a part time, which I think I have good experience doing such tasks for few years now, but I was marked with negative points, because I used to use try..except for each piece of data entities, to avoid my code being stopped if data is not allocated to one of the fields ... but the interviewer, saw that as a negative point !! at the end I was happy not being hired 😅

[wangdanny]

the cover photo is fierce. You look good John

[RS-Amsterdam]

Crystal clear like always.
Thanks for sharing !

[EnglishGeekWahoo]

Thank you very much dear John, very clear and very very helpful!!

[alitir]

Simply Genius - thx for another great tutorial - can you please add the Network error handling, logs & Sleep time....as you've mentioned, in the next one

[DrJohnnyStalker]

Hi John, i like your Videos! Iam advanced in scraping and still get good value out of many of your videos. Thx for your great channel! One thing I think would be a great video playlist is how to do broad crawls (unbounded domains) with Frontera in Python with a multi worker setup (e.g. kafka). All the best

[whatsnext]

WOW! Your channel is really growing! Congrats! I'm wondering how you never seem to run into captchas. I was making an eBay automation bot for something but would run into captchas. 😂 Keep up the good work! 👍

[SunDevilThor]

I’d love to see a video on sending POST requests to a server for real-world use.

[BokulKabir]

Hey John, I really love your videos. Can you make a video about, how to scrap any websites full table and some data from text? Then save to cvs file.

[stewart]

John - 10/10 again! Not sure why I always seem to have to add ( newline =''' ) or the CSV records are double spaced? But this fixes that issue:

with open('products.csv', 'w', newline=' ') as f:

[iamkian]

Why do my messages always get deletete every time?
I have finally managed to scrape a side after many hours of learning from you.
I needed Python and selenium to do so.
I would like to thank you so much for all your videos. They really helped me a lot!
Once again. Thank you John!