Information retrieval in the World-Wide Web : making client-based searching feasible

Finding specific information in the World-Wide Web (WWW, or Web for short) is becoming increasingly difficult, because of the rapid growth of the Web and because of the diversity of the information offered through the Web. Hypertext in general is ill-suited for information retrieval as it is designed for stepwise exploration. To help readers find specific information quickly, specific overview documents are often included into the hypertext. Hypertext systems often provide simple searching tools such as full text search or title search, that mostly ignore the “hyper-structure” formed by the links. In the WWW, finding information is further complicated by its distributed nature. Navigation, often via overview documents, still is the predominant method of finding one's way around the Web. Several searching tools have been developed, basically in two types: • A gateway, offering (limited) search operations on small or large parts of the WWW, using a pre-compiled database. The database is often built by an automated Web scanner (a “robot”). • A client-based search tool that does automated navigation, thereby working more or less like a browsing user, but much faster and following an optimized strategy. This paper highlights the properties and implementation of a client-based search tool called the “fish-search” algorithm, and compares it to other approaches. The fish-search, implemented on top of Mosaic for X, offers an open-ended selection of search criteria. Client-based searching has some definite drawbacks: slow speed and high network resource consumption. The paper shows how combining the fish search with a cache greatly reduces these problems. The “Lagoon” cache program is presented. Caches can call each other, currently only to further reduce network traffic. By moving the algorithm into the cache program, the calculation of the answer to a search request can be distributed among the caching servers