An Open Distributed System for Web Information Indexing and Searching based on Web Views

The task of indexing and searching information in the World-Wide Web face scale problems with Web size and growth. We argue that this is due, in part, to the centralized nature of Web indexing systems currently in use. This paper proposes an open distributed Web View-based system to deal with these problems. The concept of Web View is introduced, the system architecture is presented, and it is shown how the system is able to effectively build, store and search distributed indices. The components of the system modules are then described, and a working prototype is presented. 1. INTRODUCTION Two basic strategies are currently used to build World-Wide Web (Web) indices: manually updated resource lists (e.g., [Yahoo!]), and mechanically updated index bases (e.g., [Altavista] and [Excite]). The available implementations of both strategies are essentially centralized as their processes of collecting information, and storing and searching indices are performed into/from a single database or ..