Item Familiarity Effects in User-Centric Evaluations of Recommender Systems

ABSTRACT Laboratory studies are a common way of comparing recommendation approaches with respect to different quality dimensions that might be relevant for real users. One typical experimental setup is to first present the participants with recommendation lists that were created with different algorithms and then ask the participants to assess these recommendations individually or to compare two item lists. The cognitive effort required by the participants for the evaluation of item recommendations in such settings depends on whether or not they already know the (features of the) recommended items. Furthermore, lists containing popular and broadly known items are correspondingly easier to evaluate. In this paper we report the results of a user study in which participants recruited on a crowdsourcing platform assessed system-provided recommendations in a between-subjects experimental design. The results surprisingly showed that users found non-personalized recommendations of popular items the best match for their preferences. An analysis revealed a measurable correlation between item familiarity and user acceptance. Overall, the observations indicate that item familiarity can be a potential confounding factor in such studies and should be considered in experimental designs