How to make a Multi-Threaded WebCrawler in Java

Have any questions? Ask them in the comment section! Most of the time I will answer them immediately or else within 1-2 days.

Leave a Like and Subscribe!

[ThomasAllen-ob]

This was super informative! By the way, have you tried using HasData to scrape content for your crawler?

[oniecram]

The fact that it's maven compatible is just awsome
xD

[carlkarama]

Does the this keyword pass in an instance of the WebCrawler class into the Thread constructor at 5:41s line 17? I couldn't quite understand your explanation. Cheers in advance

[twilighttraveler]

How would I store the addresses in a SQL database and have a search for a link for a user?

[hamadawan]

Can you tell me which compiler you used and you create a folder multi_threaded webcrawler and in this folder you write no code but instead of of this you write the code in other folder mtwebcrawler and in this there another two files main.java and webcrawler.java I am confused I just ask if I create a folder in Visual Studio Multi-threadedwebcrwaler name and in this folder I create that two files main.java and webcralwer.java so that's work or not

[charatkakkar]

how did using Join not kill the parallelization?Join would wait for 1 thread to finish before continuing the for loop

[SHINDEMRUNAL]

Could you pls tell the time and space complexity of this code.

[skyze]

Hope someone sees this, I'm getting an multiple errors but the main one says "did not find balanced marker at 'href', does anyone know what this means and how i can correct it ?

[aqsatariq]

Can you tell me how to make a crawler to get malicious links from dom?

[mellocaster]

Bruh, where's your IntelliJ setup

[rohithkrishna]

How many webpage did it detects and stores