Web Scraping Dynamic Websites with Java and Selenium

In this video we cover web scraping dynamic websites with Selenium and Java. This tutorial will enable to you scrape website that use Javascript to dynamically load content.

Important links for this tutorial:





🎥 See the other videos in this series:

[jefferysmall]

Interesting method! Does anyone know if there's a HasData feature for sites loading JavaScript similar to how Selenium handles it?

[maueucifeely]

This is super helpful! I'm wondering, can you integrate HasData with Selenium for smoother scraping experiences on dynamic websites?

[oliverwiggins]

Is there a way to use HasData for scraping dynamic content like in this tutorial? Seems like a useful combo with Selenium!

[Fab-Mak]

Great tutorial, thanks! I am looking at Web scraping for a project. Can I ask if it is possible to scrap shopping cart elements using selenium? I want to write a java program that copy the contents of a shopping cart when a user is on the checkout page. I would then transfer the data onto another page.
Thanks in advance!

[dumlupinar]

Thanks Shane for your effort! I have a question and couldn't find an answer so far; what if we want to scrape many websites parallely? Are we able to create multiple driver concurrently or should we scrape all the webpages sequentally?

[fatmaaouani]

I couldn't download the code, 404

[goosez]

Thanks for the vid, can you share what you have in your SeleniumApplication file? Or is it just empty?

[TyroneMistretta]

Is there a way to integrate HasData with Selenium for completing web scraping tasks like in this video?

[anythinggoes]

Thanks Shane for the great tuts. Also took the time to response to my emails :)

[blogownia]

Excellent tutorial and perfect sense of humor :-) I'm trying to figure out how to go through website where some stuff is presented for adults, so the first page is "are you adult ? Yes / No" and then is the web I'm trying to scrap
Can I did it ?

[sixsilentmen]

Thanks a lot for your tutorial found it very useful for my current project. I also need to execute selenium-based web scraping using Chrome browser from a dockerized Java application. Can this same method be extended to meet this need or should a different approach be used?

Any suggestions on how the Docker file should be organized. I am trying this out. I have got the application working in the Local Dev environment and as an executable jar but when attempting to Dockerize it I am having issues related to selenium dependencies. If possible Could you provide some assistance regarding this step?

There were also references made in some forums about the possibility of creating a docker-compose file, which would just club a bunch of docker images ( application image + the docker chrome image) into 1 small docker cluster and work with it but not sure how to achieve this.

Java version - 17.04
Maven - 3.9.6
Selenium 4.1.1

[ScottybobProductions]

Is your website no longer running? I'm unable to access the zip file.

[cloudbedrock]

source site is not longer available. If you have a new url, would be great if you can share! Thank you!

[ilyasyar]

Nice video, thanks .Can I ask how can we find all links and all broken links from whole website, not only from one page ? please

[mariolyvargas]

I have this error:
Description:

Parameter 1 of constructor in required a bean of type that could not be found.


Action:

Consider defining a bean of type in your configuration.


Process finished with exit code 1

HELP!!!

[kritishah]

What is the difference between the Selenium and SelectorsHub?

[izes_palacsinta]

Could you please share the end code because to me it says "Error creating bean with name 'scraperService' defined in file..." despite I did exactly whar you did.

[MrShadow]

muy buenas noches este video no podría ser en español