filmov
tv
Using Python to find all off site links during web crawl

Показать описание
To crawl a site you look for all on-site links, but what about if you're looking for off-site links? Simple, you just invert the process. I made it its own separate function in pipulate, but I'm reconsidering that and thinking I may use the original crawl function and just return separate lists of onsite and offsite links. It will be more efficient to not have to fetch pages again to find offsite links.