How to Code a Web Crawler using NodeJs

Just a nodejs tutorial on how to build a web crawler using cheerio js and node-fetch.

------------

I'm a full stack web developer who has been in the industry since 2013. There is a lot of things I have learned along the way and I'd like to share that knowledge with anyone wanting to learn!

like this video if you found it useful and would like to see more videos of the same content.

subscribe to my channel if you are trying to improve your abilities as a web developer, software engineer, or even if you are just learning to code.

Don't forget to turn on those bell notifications!

Book mark these links!

[notalotaboat]

Incase you are struggling with unauthorized access in VS code, run it in administrator mode (right click) then run Set-ExecutionPolicy Unrestricted in terminal

[Ebizzill]

im getting a "Cannot read property pipe of underfined",  
when i console.log out results.body im getting underlined,  
what should i be looking for?
a property with the image url path ?

[mr.badanalogy]

I was following along with this tut, and for whatever reason I was having issues with url implicitly having an 'any' type. I was also having an issue with the import * as fetch from "node-fetch" for whatever reason it was saying that fetch(url) wasn't a callable. I was able to fix both of these issues in roundabout ways, but I was wondering why you didn't run into these issues in the tut.

[rinkoshopper]

really appreciate you for the unique stuff that you cover

[PodDropBits]

Subbed! Really appreciate this and the effort you put into this. Made it simple and straight to it.
I am a really "noob" with JS in general, but I did manage to do whatever you told and made it work.. The thing I want to do is something a little different but maybe easier? (I just can't wrap my head around it) the thing I'm trying to do is.. Crawl a website -> getting the "title" and the links on the specifc page of every single page and store them in a sheet/array like "title, specific links" etc. on each line for each page crawled. How would I do this?

[Ebizzill]

how do we do the results.body.pipe using axios?

[lpgmorton]

Thanks for this, Cody. I hadn't used Typescript before, and I'm not sure where it comes into play here.