How I Scrape JAVASCRIPT websites with Python

Demo of the Render() function

How we can use requests-html to render webpages for us quickly and easily enabling us to scrape the data from javascript dynamic websites. this example demos scraping video titles and links from my YouTube channel

-------------------------------------
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------

Sound like me:

-------------------------------------

Video like me:
-------------------------------------

PC Stuff:

[deepak]

YouTube algoes improving day by day. Recommended Exact video for problem I'm facing. I haven't search anything in YouTube. Bdw your videoes are always helpful. Keep it up ♥️

[vipergx]

THANK YOU!
this video crowned a half-day search for correct javascript rendering.
the magic solution was 'sleep=1' inside the render quotes..
my parser didn't get the correct output as it was needing to wait a bit for some search results.

thanks again!!!

[SeamusHarper]

Hi, if you are looking for a follow-up topic: How can you actually use js on the page, like generate data by using on-click events, or call a js function that does stuff on the page?

[NeilSpeed]

Really liked this one John. How do you usually decide if you want to go for the Javascript approach in this video vs the Selenium approach?

[rayhanferdous]

it was extremely helpful that I couldn't stop myself to say thanks.

[wall]

Hi John! Ty for tutorial, but i have a question...
I tried it but i can't get the full html code from the page i want to scrape.
There are some hidden strings which i can see when i inspect the page, but python doesnt get them :/
How can i fix it?

[venivini]

In the case where you need to click on a button to show all the elements, how can you do it with this library ? Awesome video nonetheless

[andrewn]

Nice! Thanks for this. I had no idea requests-HTML existed.

[python]

Nice video - no fluff or filler! (Or any need for bs4!)
Keep them coming : )

[DM-pypj]

I have been enjoying your videos. How is it I have only discovered them recently?!!! Anyhoos, I have watched your splash, selenium and now requests-html videos regarding handling rendering. Developer to developer, I would be interested in a video outlining the pros and cons, or a compare and contrast. When to use one or the is your preferred and why?

[noelcovarrubias]

How do you deal with no headers? When I try scraping walmart with Selenium I just can't because it will ask me to Identify my identity.

[el_bisil]

what a great explanation! you make it simple so we can understand. thank you, sir! it helps me a lot

[ashishtiwari]

Thanx for the short video. If I Interpret it correctly, then html. render basically converts the html code into a tree and then further using the .find method we search for the Id?

[davidk]

Very helpful. Glad i found your channel.

[wkowalski]

Just curious for my own learning. Why did you find and not findall when looping through videos to get the title?

[scoubydoodle]

Thanks for the knowledge. From the code in this video, I need to update alot of library before making it works. Very useful for making a bot.

[norsevelociraptor]

This works on my local machine, but I need to run this in a docker container (preferably alpine). When I do this it complains about the installation of chromium. Any ideas on how to solve this?

[rave]

Thanks! One question: If I have an Element, for example: <Element "div" class=("class1", "class2") data="100">, how can I print out the value stored in "data" (100) directly?

[wangdanny]

hey John do i need to install pyppeteer first before using requests-html?

[Jkiddoublenickle]

Would this work for pages that the href is "javascript:void(0);" ?