under the guidance of

A Focused Crawler is a hypertext resource discovery system whose goal is to selectively seek out pages that are relevant to a pre-defined set of topics. In this report, we discuss important design and implementation issues in developing the Focused Crawler as an application for the end-user. We build upon the existing architecture as discussed in [2]. Currently, the user is presented with a standard taxonomy and asked to identify one or few topics of his interest. We argue that such an interface is inconvenient to the user as he may not have a clean boundary of his interest in the taxonomy. A more comfortable approach to the user is to provide only positive example URLs and setup the Focused Crawler interactively. Based on this insight, we propose the following framework. Given a set of positive example URLs, the Focused Crawler proceeds by gathering related pages using topic distillation. These results are then prompted to the user for approval. As web pages are multi-topic and multi-zone, we plan to use techniques that segment pages into semantically coherent regions. The user can approve regions which he feels are relevant. The user can then ask the application to learn a classifier for computing the relevance of web pages fetched to the focus topic. The user evaluates the classifier and provides feedback. We illustrate the above methods using user interface screens for better understanding. Algorithms and techniques used for these tasks are also reviewed. Further, we also point to the lack of monitoring tools for the crawl frontier in the existing architecture and suggest extensions. Providing statistics about the web pages visited and unvisited, changing priorities of unvisited links are few of them. 1