Working on REAL WORLD WEB SCRAPING project on FREELANCE | Python Scrapy

Hey what's up guys, in this video I'm working on the real web scraping project for one of my clients. The goal is to extract property listings data from the sold properties, but the problem is that they are not listed on Rightmove in a casual search, so I'm looking for the property IDs within the sold houses search results and then dynamically generating links for the properties that were already sold. Finally I'm using data extraction logic created previously to scrape Rightmove.

Scraping OnTheMarket

I can't provide the source code as I usually used to because I don't have rights for that, so the only way to obtain it is to watch and follow the entire tutorial)))

💖 Become a Patron 💖

[Domsk]

This is very interesting watching you solve these problems. I am subscribing in the hope that these real life walkthroughs keep coming!

[darsanpatel]

Hi

I love your video as I am a newbie and have followed this example. I have managed to create a JSON output from the python file.
I have a question about using the JSON file to view the results of the scraper. Do I import the data into excel via the power query option? I have tried to do this, and it doesn’t seem to work as the final record doesn't have a comma (, ) so excel doesn’t recognise the end and start of each record in the JSON file.

Sorry for the newbie question.

[nicodemus]

Have you had problems with pages with JS? I use Scrapy Splash, but sometimes I get the data out and sometimes I don't