Let's Build a WebScraper in 15 Minutes with NextJS!

NextJS makes web scraping super simple. Let's dive into the easiest web scraping method, simply using the native fetch API and then turning that into HTML! All in all, building a web scraper can be done in minutes and be really helpful when there is no API you can get the data from.

Be careful with web scraping though, some websites do not allow it (especially if the scraped content is behind auth or even worse a paywall).

With that said, I wish you lots of fun implementing this into your own projects!

[codewithguillaume]

Tomorrow my friend, I'll complete this video course for sure :) Thank you !

[alberteinstein]

Love the consistency Josh, thanks for helping me learn with you.

[developer_hadi]

Thanks bro I'll watch this after finish learning react and start learning next js

[BRP-Moto-Tips]

I appreciate these kind of videos that aren't quite the bread and butter for web development but still provide insightful knowledge on some of those not mainly used topics. Thanks for your work!

[pookie]

thank you for this video ive been trying to figure out how to use next js with puppeteer for a week now this helped me

[ylovers]

Really Love the fact that you consistently put out contents that are actually very worth watching!. Josh, I have a question regarding if this can also scrap meta data of the website like title, image, subtitle and so on.

[satyarajraghav]

I was searching for this. Thanks alot.

[kwyjibu]

What should i do when deploying project. It gives localhost error, i changed it to my own domain using https etc. but still not working. Any idea?

[bigc]

Do you have the Github for this? I'm getting a 404 at the beginning.

[bryson]

This is a great video, I've been wanting to try some web scraping. Question though, why use t3 if you're not going to use trpc? I think trpc would make this more clean.

[OfficialLRY]

Could you explain why you fetch the downloads on the server and not on the client?

[doncoder-channel]

Good vide, do you have a repo for this?

[dluancr]

Thanks, that way is much better than use puppteer. My code always break when I tried use the puppeteer

[atlantic_love]

That's way, way too much work just to do something that's very simple.

[sarimkhan]

I am having error while importing jsdom library, any idea ?

[markovvoz]

Can you also provide a GitHub to this code. Thanks

[athwala]

I tried scrape data from public Facebook post but I couldn't. How can I do

[benoitgrasset]

totally useless, you can't scrap most of the sites with this technique, for example the SPAs

[OverMartinez]

What happend whe the site is a SPA and the fetch didn't get the all html from the server? there is any way to get full page content?