GPT4V + Puppeteer = AI agent browse web like human? 🤖

How to build an AI agent that can control web browser to complete tasks like research, order pizza or book flight tickets? Step by step tutorial

🔗 Links

⏱️ Timestamps
0:00 Intro of self operation systems
1:25 Market opportunities
4:20 Hubspot research report
5:31 How did AI control computer?
7:30 How does self operating system works?
10:00 Tutorial 1: GPT4V Web Scraper - Scrape anything
16:57 Tutorial 2: Web AI agent
22:45 Web AI agent demo

👋🏻 About Me

#gpt4v #autogen #gpt4 #autogpt #ai #artificialintelligence #tutorial #stepbystep #openai #llm #chatgpt #largelanguagemodels #largelanguagemodel #bestaiagent #chatgpt #agentgpt #agent #babyagi

[AIJasonZ]

What AI agent use case do you think it will unlock with browser & computer access? Comment and let me know!

[TheHeatzz]

reCAPTCHA: Are you a human?
Web AI Agent: *click*

[elevenchicago]

You have been making some of the best videos on this stuff since day one. Always super practical yet very valuable content Jason. Keep it up and glad to see the HubSpot sponser! Esketit

[avi]

Bro, you are such a gangster. You're the only person I follow actually putting out valuable work at this level. All these AI grifters are on their six hundredth basic ass autogen demo, just zero value videos for clicks.

[BrianMosleyUK]

This is really valuable work Jason, beautifully conceived and executed. Keep going, this is huge. 🙏👍

[paresh]

Great Work Jason!! would love to see you explore creating a Autogen like framework with Open source Multimodal agents, seems like it can open a lot of possibilities without the huge costs associated with using openAI's APIs for intensive tasks. Keep up the great work.!

[oshodikolapo]

This is a really smart and intutive process. Thanks Jason!

[jasonfinance]

Great job mate! Love to see all the agent possibilities

[amandamate]

AutoGen now has a AgentBuilder class. Its exactly doing how it sound, building a swarm of agents.

[redamarzouk]

I've been very excited about the last video when I first saw the paper of GPT-4v and the automation applications.
I do RPA full time and even though GPT-4v is not there yet in terms of what it can do on its own for form data entry and complex automation processes, I can see the use cases where it can outshine RPA when it's enhanced.

[EverythinTechnology]

Great video! I would love to see if this can be done with an open source model like LLaVa.

[Jim-eyry]

Nice, was wondering how to build a chrome plugin to control web like hyperwrite, this is handy 👍 Also the hubspot research is actually good

[paulmartz]

Congrats on your work, really useful content. Keep it up!

[Yakibackk]

thanks jason for this valuable work. This toutrial are advance but very valuable spec in the market. You could easliy make big money from this method.

[akshaygoel]

Amazing work! Thanks for putting in so much effort to bring quality content to the viewers.

[giovannidamico]

Great work Jason! Been looking for examples of scraping with screenshots for a while! I am just wondering what are the costs implications compared to traditional scrapers

[exaliber]

Hey Jason amazing video as usual! Can you do one using canva to setup and automate a youtube from creation of videos using canva to scheduling the posts... do you think this will be possible?

[leewsimpson]

Nice work- thank you!!

Do we really need to run node? Can’t we just use python browser libraries ?

[JoeyZero]

Just throwing this out there: If using OpenCV to drive your annotations, you can annotate coords directly on the image without too much difficulty (to give some guidance for when GPT4V is trying to recommend a location to click). Haven't done this with puppeteer, but I'm sure it can be done ;)

Anybody wanna collab on some experiments in this space? 🙏 Lolol

[youwang]

Jason, wondering if you have some kinda plans for diving into details of finetuning open source models like llama2?