Always Check for the Hidden API when Web Scraping

If this method if available, its the best way to scrape data from site. I will show you how to find the API endpoint that we can use to directly get the JSON data that is being sent from the server, before JavaScript gets its mucky paws on it and makes it look like what we see in our browser. Its quick and simple, and with just a few extra tips and techniques it can transform your web scraping.

[hectord.]

I've doing this for years as a self taught programmer, there are some little tricks you did here that i didn't know, thank you for the video.

[Davidca]

This single-handedly cut the running time of my program from literal hours to a couple of minutes, cannot thank you enough!

[leow.]

I'm not a very experience programmer, I've been doing it recreationally for like 2 years on and off but I did a lot of webscraping and this is just a really neat piece of knowledge that I wouldn't have come across own my own. Thanks.

[abc.]

I've been using this trick for a while now, and I've learned it from you, so thanks. Amazing work man

[zenon]

Please ignore my first comment. I checked out your first video in this series and learned about using scrapy shell to test each line of code. With that I found the bug in my code. The code worked PERFECTLY as advertised. Your the man! Much thanks!

[angestellterderantifagmbh]

My man, this is EXCACTLY what I was looking for. Had to do some extra steps, but a little try-and-error and basic understanding of HTTP was enough to solve my problem. Thank You!

[huisopperman]

Thanks for sharing! This has helped me a lot. After struggling for weeks with selenium, I was able to apply this technique fairly quickly, and am now using it as source to scrape ETF-composition data to feed directly into a PowerBI dataset. Much appreciated!

[freekdl]

I rarely praise anything, but this tutorial was SO good! Well explained, no filler. In 7 or 8 minutes you guided me through finding the hidden information I needed, which tools I need to use and how to automate it. This tutorial gave me enough confidence to try to write my first Python script! Within hours I built a scraper that can pull all metadata for a full NFT collection from a marketplace. Without this video it would have taken days/weeks to discover all of this

[transatlantc]

Nice video. It’s worth noting as well that many APIs will pageinate, so rather than checking how many total results exist and manually iterating over them - you just check to see if the ‘next page url’ or equivalent key exists in the results and if so, get that too until it doesn’t exist anymore, merging/appending each time until the dataset is complete 👍

[judgewooden]

I like how you regularly start sentences with 'you might think' assuming we are all idiots. I approve, glad smart people, like you, make time to explain to us plebs how the world works. Apprecated.

[Kralnor]

This is a true gold nugget. Thanks for demonstrating how to easily view the request in Insomnia and auto-generate code!

[drkskwlkr]

Loved everything about this video! Great delivery style, production quality and interesting topic for me. First time visitor to this channel and not a Python user (thanks, YouTube, for your weird but helpful predictive algorithms).

[shivan]

This is, no joke, the most useful video I ever saw on Youtube!

[flfywnk]

There is always something new to learn.

I’ve been spending hours to grind such an information by hand-writing the whole program to get my result ;D
Thanks!

[tikendraw]

I just want you to never stop creating such informative video. For god sake.

[fuad]

really nice and helpful tips in an actual topic with a sight-pleasuring recording quality, thank you for your time and efforts.

[anduamlaktadesse]

Hidden API is by far the easiest way to scrap a website!!!! Thanks bro!!! Big Clapping for I've followed all your procedures &finally i did it.

[krahkyn]

This is such useful content that shows how much value experience gives — thank you for the straightforward and realistic tutorial!

[mujeebishaque]

I love you, John. You're awesome! Thanks for being unique and producing quality content.

[ERol-durd]

Awesome advice, a lot of people skip checking the requests when building scrapers but it can save a lot of time when it works