Rajiv Gandhi college of Engineering and Research

Abstract – As the number of Internet users and the number of accessible Web pages grows, it is becoming increasingly difficult for users to find documents that are relevant to their particular needs. The key factors for the success of the World Wide Web are its large size and the lack of a centralized control over its contents. Users must either browse through a large hierarchy of concepts to find the information for which they are looking or submit a query to a publicly available search engine and wade through hundreds of results, most of them irrelevant. Web crawling is the process used by search engines to collect pages from the Web.Web crawlers are one of the most crucial components in search engines and their optimization would have a great effect on improving the searching efficiency. This paper, introduces web crawler that uses a concept of irrelevant pages for improving its crawling performance. Despite their conceptual simplicity, implementing high-performance web crawlers poses major engineering challenges due to the scale of the web. This crawler computes the weights for the pages we come across during the crawling process and hence decide how much a particular page is important to us. Both issues are also the most important source of problems for locating information. The Web is a context in which traditional Information Retrieval methods are challenged, and given the volume of the Web and its speed of change, the coverage of modern search engines is relatively small. Moreover, the distribution of quality is very skewed, and interesting pages are scarce in comparison with the rest of the content. Keywords: Web crawler: A web crawler is a program that, given one or more seed URLs, downloads the web pages associated with these URLs, extracts any hyperlinks contained in them, and recursively continues to download the web pages identified by these hyperlinks