Efficient information retrieval from the World Wide Web

grantor: University of TorontoWith the explosion of information that is currently available through the World Wide Web (the Web), searching for and retrieving anything has become an increasingly daunting task. For this reason, many different search engines such as AltaVisa, Excite, HotBot, InfoSeek, Lycos, Yahoo, etc. have been built to help Internet users. These search engines employ a program called Web Robot which searches and retrieves any new or updated documents from many Web sites, then passes the retrieved documents to the index server. The Web Robot is also used for other purposes such as statistical analysis and mirroring Web sites. The wide popularity of using the Web Robot resulted in increased usage of the limited Internet resources, especially the net-work bandwidth. Because of this, the results that are delivered by the Web Robot are far from satisfactory. In this thesis, we first discuss in detail the pros and cons of using the Web Robot. Then we introduce the Distributed Information Gatherer DIG, a system that provides an efficient and effective method of continually gathering information from the Web, as a way to improve the efficiency in utilizing the limited Internet resources. Using the DIG specifications, we present the results of an experiment to analyze the utilization that could be achieved through the deployment of the DIG over the Internet. Finally, we generalize the ideas behind the system and discuss the deployment issues of this system.M.Sc