The influence of recall feedback in information retrieval on user satisfaction and user behavior

Contains fulltext : 141626.pdf (author's version ) (Open Access)Users can judge the precision of an IR system by inspecting the retrieved documents. They cannot judge recall, however, which would involve inspecting the whole collection, thus obviating the IR system, and impossible in the case of WWW. How then, can we ascertain whether recall is a valid metric for the quality of an IR system as perceived by the end-user? In a carefully controlled experiment we presented users with a simulated web search engine. Besides the search results, the engine could give a (spurious) recall estimate, presented as a pie chart. We manipulated this recall feedback and whether the information need was fulfilled with respect to quantification type (the number of documents requested). The results reported here suggest that whereas recall may be a valid metric for designers and evaluators of IR systems it may lack validity as a metric for search quality as perceived by the end-user