Easiest SELENIUM Web Scraping for JavaScript Pages

Modern web scraping is all JavaScript! Learn another method of extracting data from a JS website by using Selenium to load the page and grabbing the rendered source HTML. its really easy and can work very well in certain situations. It's all about knowing the tools available and using the right one for the job at hand

-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------

Sound like me:

-------------------------------------

Video like me:
-------------------------------------

PC Stuff:

[leemeng]

Thanks for the clear instructions and demo. I especially liked how you first showed what WOULDN'T work. Sure saved me some time!

[manafbargash]

omg, you are AMAZING! Please keep doing what u doing. You show us stuff that takes hours and hours on Google to find and sometimes can't even being found. Thank you!

[celerystalk]

I've never used beautifulsoup together with helium/selenium. What a cool tip! Thank you as always John!

[alexcrowley]

Very rarely ever comment on yt vids, but your content is unreal, thanks mate from a fellow webscraping Brit

[adnanyounas]

Thanks mate!!! Really appreciate your efforts
Continusly seeing and practising your concepts from whole night....
Respect!!

[evalaviniabucur]

Thank you! I was really looking for something that would allow me to scrape pages generated dynamically with javascript and I was getting nowhere. I know about selenium (which, for some reason, didn't work for me), about PhantomJS (which was no longer supported or was not able to do its job anymore when I tried it or something). I remember I succeeded somehow (it was through the use of ChromeDriver as well, but I was getting annoyed because, when running, the Chrome Web Browser window would open before getting the results and close) but I gave up and stopped trying to scrape pages with javascript generated content. Your approach runs the ChromeDriver in the background - which I find extremely pleasant. The only two things I'd add ('cause I myself bumped into them) would be 1. ChromeDriver's location HAS TO BE in the Path, 2. ChromeDriver's version should correspond with the version of Chrome one is using on his machine. Other than that... a great tutorial and thanks, again, a bunch! ^_^

[rtxmax]

This is much better than selenium, I was using selenium with the geckodriver and the output html was not showing the dynamic html text, but with helium it does.

[kaoutharmokrane]

Thank you so much for the incredible content you're putting out here for free.
Subscribed!

[topkek]

wow, actually a tutorial that is on point for once. thanks a bunch!

[ArchIntelLTD]

Thank you very much. I was able to scrape a Javascript website

[ericxls]

really good video... took me a very long way! got stuck with the headers, but nothing that chatgp did not solve in seconds!

[maryanpetryshyn]

Thanks man. It was short and useful video.

[nusratjahan]

Thank you soo much. It helped me :D.
Keep growing. xoxo

[tithikulkarni]

Thanks allot for making this vedio and sharing this trick u are the savior!

[liljack]

Damn bro you save me sooo much time thank you so much!!!

[peterkim]

Thanks john. Can you do a tutorial on how to scrape airbnb's listings urls

[tnssajivasudevan]

This is going to be useful for my project. Thank u Sir

[humushaikh]

Hey John,

Thanks and a million of them to you.
I was looking out for something similar for running the chrome in the background
Could you also help me with a command to embed in the code where we can also stop the headless chrome once the job is done, i meam i want to kill that chrome instance after every iteration.
Await your response.

Thanks again with lots of love.

[eziola]

Have you run into JavaScript executing Shadow DOMs? I had to make a JavaScript function that navigated the nested shadow DOMs and returned the "bottom" shadow DOM HTML to beautifulsoup 🙄

[sachinbhagat]

Can you please post or give some idea to scrape tradingview chart indicator values