A Turbocharged Web Crawler built on V8 ECSE 420-- Parallel Computing Group 1 Fall 2011

Our group designed and implemented a web crawler this semester. We investigated techniques that would allow us to fetch a webpage, gather the URL links on the page, and recursively repeat this procedure for all URLs discovered, storing our results in a graph data structure, with nodes representing web pages, and directed edges links between these webpages. This is by no means an original endeavour; there are many companies using web crawlers, including Google, Microsoft, Baidu, and others. There are many different reasons for wanting to crawl the web, from commercial to private. One might want to index all content on the Internet, or create a graph of the interconnections between webpages, for the purpose of searching for content on the web. Crawling the web is a problem which is highly parallel in nature. The sheer size of the Internet, and the volume of content online is reason enough to investigate parallel solutions. Furthermore, there is no clear starting point, or ordering of the webpages, rather the simple requirement that they should all be fetched and parsed exactly once. Using a simple graph search algorithm such as breadth-first offers a naturally parallel implementation pattern. Any efficient web-crawler must exploit parallelism, and in that regard our solution doesn’t distinguish itself either. There is a clear indication that the process of fetching webpages, parsing the