Lessons learnt from Web Scraping 9 billion pages

The process of web scraping usually involves spiders that fetch the HTML documents from relevant websites, extract the needed content based on business logic, and finally store it in a specific format. But there are numerous challenges if you are scraping data at large scale. Cathal Garvey in this video talks about his experiences scraping 9 Billion Pages/month.

[anandc]

If I try to fill in the form given at the bit ly link to get the full video, I get an error message.