Automate Chrome in an API with Puppeteer and Netlify Serverless Functions

Learn how to use Puppeteer to automate headless Chrome inside of a serverless function. We'll use Netlify to create a new API endpoint where we'll automate scraping a page's metadata and interactive search results.

🧐 What's Inside
00:00 - Intro to automating Chrome with Puppeteer and Netlify functions
00:10 - Tools we'll use and what to expect
01:00 - Starting a new node project from scratch
02:20 - Installing the Netlify CLI and logging in
03:53 - Creating a new serverless function with Netlify
05:32 - Installing Puppeteer and Chrome dependencies
06:50 - Configuring and launching a new browser locally with Puppeteer and Chrome
10:11 - Scraping a page's title and description from Chrome with Puppeteer
11:50 - Making a clientside search and scraping the results with Puppeteer
13:36 - Deploying serverless functions with Netlify CLI
15:43 - Outro

💾 Code

🗒️ Read More

🎬 Next Up

🔔 Subscribe for more tech and developer videos

🐦 Get updates straight to your Twitter @colbyfayock

📸 Catch the next stream live on Twitch @colbyfayock

✉️ Or a newsletter right to your inbox!

💝 Sponsor me for more free content like this!

🎥 Want to know what A/V equipment I use?

🧰 More Resources

puppeteer-core

chrome-aws-lambda

Puppeteer

Netlify CLI

🎼 Music
Music from Uppbeat (free for Creators!):
License code: TLACVOID0A8C0LTV

#colbyfayock #puppeteer #serverless #netlify #chrome #api #webdevelopment

[robinbeaudru]

Thank you man! Amazing!
We want more of those Puppeteer serverless stuffs!

[MaxProgramming]

This is exactly what I wanted. I am loving netlify serverless functions and puppeteer but i wanted to know if they can work together

[TKTBC]

Thanks for 1000 times hahaha I've used a couple days to find solution for using puppeteer with serverless functions. Now I found it on you video

[christiandiazportela]

Thanks a lot, Colby! your channel is underrated! regards from Bolivia.

[c]

Very useful and interesting as always, thanks colby :)

[mariusailisoaie]

Thank you very much for the tutorial! It was useful and helpful.

[k.t.]

This tutorial was very useful. Thank you

[dragosvlasie]

Great video! And thanks for sharing the github link as well

[muneebakram]

Awesome man, You are great. Thanks for sharing.

[bassguitarist]

Thanks for the tutorial, I get all the way to the point where I test to return the title but I get this "message": "Protocol error (Page.navigate): Target closed.", I put await on mostly everything that I can see would need to.

[rushdude]

Very helpful video, although I am struggling with the deployment stage. When running netlify deploy, it just seems to hang until eventually I get a JSON HTTP error with a status code of 500 and the logs on netlify just say cancelling deployment and I'm not sure why. Following your example, I don't have a build script since my app runs the same as yours, however it seems like when attempting to deploy from Github instead of the CLI, netlify runs a build command, which we don't have in this case.

[seewai]

Do you have any guidance on setting this up with CORS? Right now, when i try and pass data in the body of the request or just try and ping the endpoint, I get a CORS error and the function doesn't work. This happens on localhost and live.

[justinfleagle]

What if I have to sign in using Puppeteer, snd I have to keep the connection open to wait for a user of the application to make API requests?

[fadelibrahim]

what about netlify functions pricing? can you give an example how it is gonna be if we scaled scraping puppeteer with netlify how much does the free plan will last

[jamesgrubb]

Hi, Colby thanks for this tutorial, fantastic post. I'm trying to port this workflow to Nextjs. Firstly I tried as is and was getting a bunch of errors I then tried using puppeteer (as opposed to puppeteer-core) this worked locally but failed when uploaded to Netlify and Vercel's servers. Any thoughts, please.

[kizhissery]

how to use context to change the variable in url pls

[cjaffiliated]

can you do puppeteer web scrapping video for scrapping aliexpress product page, the product description

[eleah]

Team! Can I outsource my brain to puppeteer? I think I can.

[jazperruedas]

Hey really awesome work!! I'm still new with back end stuff, I really wanna know if I can implement this to my chrome extension, I want it to show the data on other website(adding element div using chrome extension stuff) using headless browsing but need it to be interactable like "VehicleStatus("DJAI", "3C6MRVJGOME571321") " and will pull up the status on the spefic page I wanted

[staz-tech]

Does anyone else get this error? "errorMessage": "Failed to launch the browser process!