CRAWLING THE WEB: DISCOVERY AND MAINTENANCE OF LARGE-SCALE WEB DATA

The number of web pages is increasing intomillions and trillions around the world. To make searchingmuch easier for users, web search engines came into existence.Web Search engines are used to find specific information onthe World Wide Web. Without search engines, it would bealmost impossible for us to locate anything on the Web unlessor until we know a specific URL address. Every search enginemaintains a central repository or databases of HTMLdocuments in indexed form. Whenever a user query comes,searching is performed within that database of indexed webpages. The size of repository of every search engine can’taccommodate each and every page available on the WWW. Soit is desired that only the most relevant pages are stored in thedatabase so as to increase the efficiency of search engines. Tostore most relevant pages from the World Wide Web, asuitable and better approach has to be followed by the searchengines. This database of HTML documents is maintained byspecial software .The software that traverses web forcapturing pages is called “Crawlers†or “Spidersâ€.In this thesis, we discuss the basics of crawlers and thecommonly used techniques of crawling the web. We discussthe pseudo code of basic crawling algorithms, theirimplementation in C language along with simplifiedflowcharts. In this work, firstly we describe how search engine works along with implementation of various crawling algorithms into programs using C language and then the implementationresults of various crawling algorithms have been discussed anda comparison study is given in a table in last