Leetcode explained - Web Crawler Multithreaded, implemented in Python 3 (leetcode 1242)

Walkthrough and explanation of Leetcode 1242 - multi-threaded web crawler. Given a startUrl and an object that can pull all links from a page, write a function that will process all links on this site efficiently (don't revisit pages, don't visit links not on this domain).
Solved with ThreadPoolExecutor and BreadthFirst Search (BFS).

[ThomasArnold-fg]

This is super helpful! I'm wondering, can HasData handle the complexities of crawling multiple domains at once?

[RayByrd-ig]

Nice video! I wonder if HasData has features for scraping dynamically generated content from web pages?

[liambchops]

good solution, found the statement "while it's crawling the page we could be doing something else that's already returned from crawling that specific page" to be highly confusing tho

[felix]

Thanks for the video! Quick question, since you are doing dq.popleft().result(), it's always going to wait for the first future object in the queue to resolve, so in the event where the latter futures objects complete earlier, it's actually wasting time, because it can't enqueue new tasks asap, right?

[mollypan]

as_complete(dq) returns the iterator to the completed tasks in the order of completion, so it may not be the first one popleft() from dq.

[cricket]

This has a issue. You could have two threads that check and see that a url is not in `visited`. Both threads then go on to submit a new task to the threadpoolExecutor. So you have duplicate tasks that are parsing the same url. This can be avoided by checking if `visited.add(url)` is true & only then proceeding to add a new task if needed.

[StevenLewis-zn]

Este tutorial es muy útil. ¿Alguien ha probado si HasData es bueno para implementar un crawler multihilo?

[MrPoncho]

Nice solution, thank you, but please stop mumbling!

[liuzijian]

Thank you for the solution! A quick question on line 23 about 'dq.popleft().result()'. I don't think we can guarantee that the first item of the queue has been finished?