Crawl and Follow links with SCRAPY - Web Scraping with Python Project

Scrapy is a powerful web scrapign framework for Python, we can use it to following links and crawl a website, in this case I am going to scrape product data from an online store following each category link one by one and export the data to a CSV file.

Support Me:

-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------

# Timestamps
00:00 - Intro
00:30 - Startproject & genspider
01:30 - Website categories
02:25 - Scrapy Shell

05:15 - Follow Links
07:15 - Parse product data
13:33 - Scrapy Crawl
14:15 - Results CSV
15:05 - Summary & Outro

[hmak]

The guy who left a dislike think you misspelled "scrape" as "scrapy"! :)
Otherwise, who is that dumb enough to dislike such informative content?

Thanks for all your help, Mr Rooney. You are the reason why I passed an interview just a few days ago!

[tubelessHuma]

Well done john. Need more Scrapy Tutorials.🌹

[mokolokoa]

The best scrapy thing i ever seen. Great Work !

[vickysharma]

I was looking for this kind of video, executing with selenium for this task took more time.
Thanks for the video Man.

[dickyindra]

WELL DONE!!!! GREAT VIDEOS SIR JOHN!!!!

[samjane]

this is insane work man keep it up and smile more please lol :D

[GelsYT]

literally answered every question i have

[vladimirantonov]

Thanks a lot! Very interesting and informative!! :))

[rafewm]

Thank you that was really useful <3

[omidasadi]

great, thanks for sharing it....
John how could deal with pages without pagination? I mean pages with a policy of scrolling down and appearing continuous products.

[bitarddrag]

Hi John thanks cool video! i have a question is it possible to somehow make scrapy go check whether this category has subcategories and go through them and parse and if not then parse what is there if you know then explain how to do I will be very grateful and apologize for the language barrier

[vishalverma]

Hey John ! I guess you missed some thing here, what about pagination how should I apply in such code ?

[umarsaid]

What if the categories spans to multiple pages? So I want to follow the next button and for each page found, I want to follow the links to each detail item. I can use CrawlSpider, but I wonder how to use Spider only.

[CenTexCollector]

Site I'm looking like doesn't have all the CSS just HTML it looks like. Do you have a tutorial video on getting responses for that?

[palashpathak]

hi John, i am getting 503 error when i do scrapy shell 'url', i have tries setting the user agent, still same, can you help please

[ulfgj]

so, when we want to follow links in ALL THE underlying pages that has a specific url pattern - not just one level down - this isn't it, right?

[hayathbasha]

Hi,
Please advice me on how to improve / speed up the scrapy process

[amritanshusharma]

Hi, i watched your video and it's awesome, so i am also doing some project just like you did on this video but in different way and i stuck. Here is what i want, first i want the data from each column then open the link and then extract some extra data from that link . I know it sound crazy but that's the project i am doing right now using scrapy. I check scrapy documentation and i didn't find anything similar to what i want.

[melih.a]

Is there a way to have the default website a link from playwright after logging in?

[youvanced]

What are the benifts of using this instead of the Requests library?