PERFECT REQUEST OF ON DEMAND WEB SCRAPING TUTORIAL | Multi level recursive site crawling | SCRAPY

Hey what's up guys, here's an example of perfect FREE web scraping tutorial request - the range of links to crawl as well as features to scrape are specified precisely so I don't need to waste time on guessing what to scrape.

Thanks to Devi Nanda for providing such a nice topic for a video!

Video source code
(pagination works but not safe)

Improved pagination crawling source code
(please use this version in the production)

[yogiaryavrat]

thank you for the advanced tutorial. unique in a way

[aarons]

Hi Maksim, hope you're keeping well. Just briefly went through the code, had you considered using Rules and LinkExtractor to do recursive scraping ?

[devinanda]

Hi Code Monkey King, thanks for the video and code. I really appreciate your effort.
Also, on every page, the total page number of pages is already mentioned, so I guess it could have been taken from there directly saving a lot of code.
0-9, also contains data(so total pages are 0-9, a-z) so I guess string.ascii won't work in that case, anyways I have worked that out.
The way you extracted the items in the table is amazing, I used the other technique like you do while extracting features. Long process!!
Thanks CodeMonkeyKing.

[python]

Nice video! 👍 - Reminds me of your amazing Urban Dictionary code!
*Haven't used response.meta.get before - always learning from your videos - Thanks man!

[shafi]

Hi bro, i have question about how to make scrapy use threading and how i can use callback on all url in website fir example i wanna create scrapy project to crawl all URLs inside main page then use callback on all URLs to extract products name or something.