Intelligent Peer Networks for Collaborative Web Search Articles

■ Collaborative query routing is a new paradigm for web search that treats both established search engines and other publicly available indexes as intelligent peer agents in a search network. The approach makes it transparent for anyone to build his or her own (micro) search engine by integrating established web search services, desktop search, and topical crawling techniques. The challenge in this model is that each of these agents must learn about its environment—the existence, knowledge, diversity, reliability, and trustworthiness of other agents—by analyzing the queries received from and results exchanged with these other agents. We present the 6S peer network, which uses machine-learning techniques to learn about the changing query environment. We show that simple reinforcement learning algorithms are sufficient to detect and exploit semantic locality in the network, resulting in efficient routing and highquality search results. A prototype of 6S is available for public use and is intended to assist in the evaluation of different AI techniques employed by the networked agents. Centralized search engines cannot cover the entire web (Lawrence and Giles 1999) because it is too large, fast-growing and fast-changing (Brewington and Cybenko 2000; Fetterly et al. 2003; Ntoulas, Cho, and Olston 2004). As a result, current centralized search engines focus on “important ” portions of the web. However, the notion of importance is highly subjective: the biases that are introduced to address the needs of the “average” user can result in diminished effectiveness in satisfying many atypical search needs. Therefore, the “one engine fits all” model cannot handle the increasing size, rate of change, and heterogeneity of the web and its users. In addition, as search becomes more prevalent at the desktop level, users will increasingly want to make subsets of the files indexed in their computers available to others through the Internet. Peer networks provide us with an architecture for extending web search technology to capture the contextual needs of a diverse population of users, while leveraging their resources. There are several models of peer network topologies and query protocols, including structured, unstructured, flooding, distributed hash tables, and hierarchica