Assessing Contextual Suggestion

Assessment for the TREC Contextual Suggestion Track is unusual in that it depends on the personal preferences of as-sessors. During the initial phase of the track, assessors rate points-of-interests in a source city (Philadelphia for TREC 2013) in terms of their own interests. These rankings are distributed to participating groups, who are given about a month to generate point-of-interest suggestions for fifty other target cities around the United States, with personal-ized suggestions generated for each assessor on the basis of their ratings for the source city. These suggestions are then returned for rating, with each assessor rating their person-alized suggestions for one or two of the target cites. Effec-tiveness scores (e.g., precision at rank 5) are then computed from these ratings. Unlike traditional TREC tasks, such as adhoc retrieval, it is not possible to measure assessor agree-ment, since each assessor is rating each point-in-interest in terms of their own personal preferences. Instead, we mea-sure assessor consistency, which we define as an assessor’s tendency to rank systems in the same order as other asses-sors. While consistency can be quite high for some assessors, and appears reasonable for most assessors, we have been un-able to identify predictors of assessor consistency, including past consistency. 1