A Review of Web Crawler Algorithms

Abstract: The web today contains a lot of information and it keeps on increasing everyday. Thus, due to the availability of abundant data on web, searching for some particular data in this collection has become very difficult. Emphasis is given to the relevance and robustness of data by the on-going research-es. Although only relevant pages are to be considered for any search query but still huge data needs to be explored. Another important thing to keep in mind is that usually one’s need may not be desirable for others. Crawling algorithms are thus cru-cial in selecting the pages that satisfy the user’s need. This paper reviews the researches on web crawling algorithms used for searching