Hidden APIs with Scrapy - easy JSON data extraction

I've shown this web scraping method before but never using Scrapy, and given that the Scrapy framework gives us some reaslly good features I thought it was about time I demo'd this. This is it in its most basic form.

This Scrapy project will should you the basic methods for scraping API like data from a website, be it a proper API or the API endpoint you find when scraping a web site.

Support Me:

-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------

[tubelessHuma]

Good to see it in Scrapy. Your channel need more Scrapy tutorials. 👍

[isaialawaniyasana]

Awesome videos John. I wish I had found you before I paid money to learn everything you're explaining here more succinctly and free 👏

[brothermalcolm]

i requested for your scrapy x api video and voila it's right here, thank you!

[lfcatchall]

Love your videos, really helped me get a project off the ground. Could you do a video on not overwhelming an API server with requests? What is the best way to slow the requests to the server down? I would like to hear your thoughts, process, etc. keep up the great work!

[decromax]

As always, detailed & clear explanation. Threw me off when Pycharm was fired up though 😅

[wangdanny]

Hey john! Many guitars in the back. So any plans for a music youtuber soon?

[rostranj]

Could you make a tutorial where you deploy the scraper on a VPS? I've seen many options like using scrapyd or running a cron job. I'd be helpful to see examples.

[jamesmining]

seems to me every website has its own custom API and blocks access to these type of request to even HTTP GET data

[heisenbergwhite]

Loved the video!
Any plans for a web scraping course soon?

[kevinz]

super cool very well explained thanks so much. subscribed :)

[Scuurpro]

I'm getting a 429 unkown error. What type of method should I use to slow down my scraper calls?

[xiaohongchen]

That's a nice video. Like always. Hi, John, Can you post a video to show how to scrapy home depot product reviews? Thank you.

[fred_vids]

Do you have a video or can you create a video on how to schedule recurring scraping? Ie, say having the scrape run every hour?

[hirisraharjo]

Awesome! But what if the website doesn't make any xhr requests? Is headless browser the only way (by clicking and pretending to be a user)?

[renatosardinhalopes]

Hello John, could you compare Python Requests and Python Scrapy? I just found out about scrapy but want to know the caveats between the two.

[zahrastb]

Great video! The hidden api I'm trying to work around is a fetch type though. And it's response is really not as clean as this one. I don't know how to work with it really

[zheyuan]

Great Video. I am wondering if scrapy can get that long URL by itself instead of copy and paste by ourself?

[JackyVSO]

Is there a good reason to use Scrapy for this instead of the requests library? Isn't it bringing a gun to a knife fight?

[sudhanshuyaa]

Hi John
Can you please guide about instagram scraping
Thanks

[DittoRahmat]

Hi John,
I tried scraping a website using hidden API like this, I succeed in parsing the first page.
When I tried to loop the next page, it returned 403 error.
Now, when I tried going back to parsing one page only, it also returned 403 error
I have tried changing the user agent in the settings.py, but still no luck
I can open the API endpoint link just fine in browser. So I think it's not an IP ban
Can you suggest something ?