On finding needles in WWW haystacks

This paper discusses experience in retrieving specific items from the WWW. The Intelligent Agents Laboratory at the University of Melbourne has spent the last two years building programs to autonomously retrieve items such as sports scores, university subject descriptions, and paper citations. A wide variety of Web resources have been used. The approach is knowledge-based in the spirit of expert systems, where domain and task specific knowledge is crafted into general purpose shells. The key insights are the value of designing conceptual models that can mesh with physical models of the WWW, and the value of classifying domain and task knowledge. Licence is taken to place the research in the context of wide-ranging issues such as the information revolution, styles of computer science research, and the metaphor of software agents