An Approach to Large Scale Distributed Information Systems Using Statistical Properties of Text to Guide Agent Search

Introduction and Problem Statement Given a large, dynamic corpus, distributed over several (possibly hundreds or thousands of ) physical sites traditional Information Retrieval techniques don't seem to scale [BDMS94]. Web-walking robots are fine as far as they go, but they're centralized, they don't stay current, and, except for Harvest [BDHMSW95], none, as yet, have any notion of local indices that are shared. 2 A Proposed Solution The solution we propose uses a mediated agent architecture composed of local server agents managing local corpora and communicating via agent name servers and brokers [see Figure 1]. Use of this intelligent agent architecture leverages the scalability solutions from the intelligent agent community for use in distributed information retrieval. In addition to the use of a mediated architecture, our solution depends upon the use of automatically generated, effective metadata. In this case, metadata is effective when it provides accur