Users want more sophisticated search assistants - results of a task-based evaluation

The Web provides a massive knowledge source. The same is true for intranets and other electronic document collections. However, much of that knowledge is encoded implicitly and cannot be applied directly without processing it into some more appropriate structures. Searching, browsing, question answering for example could all benefit from domain specific knowledge contained in the documents; and in applications such as simple search we do not actually need very “deep ” knowledge structures such as ontologies but we can get a long way with a model of the domain that consists of term hierarchies. We combine domain knowledge automatically acquired by exploiting the documents ’ markup structure with knowledge extracted on the fly to assist a user with ad hoc search requests. Such a search system can suggest query modification options derived from the actual data and thus guide a user through the space of documents. This paper gives a detailed account of a task-based evaluation that compares a search system which utilizes the outlined domain knowledge against a standard search system. We found that users do use the query modification suggestions proposed by the system. The main conclusion we can draw from this evaluation however is that users prefer a system that can suggest query modifications over a standard search engine which simply presents a ranked list of documents. Most interestingly, we observe this user preference despite the fact that the baseline system even performs slightly better under certain criteria.