Web Scraping Tutorial #5 | Complete Scrapy Project with Infinite Scroll | How To Scrape 'Load More'

Part 5 of our latest Scrapy project / Web Scraping Tutorial. Getting the hidden text.

Web Scraping a large site with "infinite scroll" aka "AJAX load more".
(This is based on a real world project from Upwork).

*** PART 5 of 7 ***

This video covers the getting the hidden text, hidden until "click for more" is used in a browser. We get it from the json...

Yet again we can use :

This extracts json and allows us to parse the dictionary that we get from it.

Please comment, subscribe, like, as this is taking a lot of effort and the more comments/feedback the better we can tailor future videos to you.

GitHub Code

Cheers,
Dr Pi.

#scrapy #infinitescroll #loadmore