Web Crawler: A Survey

- In world wide web, Web Crawler is working as a software agent which helps in web indexing. Web crawler at times called as spider or internet bot explicitly operated by many search engines. In Information Retrieval to colleting an information Web crawler play an important role. For effective searching and web indexing is mostly depends on web crawlers. General Purpose, Distributed and Focused crawling are the types of crawling techniques of web crawling. This paper discussed overall survey on web crawlers, its types and working of web crawlers