A decision theoretic framework for ranking using implicit feedback

This paper presents a decision theoretic ranking system that incorporates both explicit and implicit feedback. The sys-tem has a model that predicts, given all available data at query time, different interactions a person might have with search results. Possible interactions include relevance la-belling and clicking. We define a utility function that takes as input the outputs of the interaction model to provide a real valued score to the user’s session. The optimal rank-ing is the list of documents that, in expectation under the model, maximizes the utility for a user session. The system presented is based on a simple example util-ity function that combines both click behavior and labelling. The click prediction model is a Bayesian generalized linear model. Its notable characteristic is that it incorporates both weights for explanatory features and weights for each query-document pair. This allows the model to generalize to un-seen queries but makes it at the same time flexible enough to keep in a ‘memory ’ where the model should deviate from its feature based prediction. Such a click-predicting model could be particularly useful in an application such as en-terprise search, allowing on-site adaptation to local docu-ments and user behaviour. The example utility function has a parameter that controls the tradeoff between optimizing for clicks and optimizing for labels. Experimental results in the context of enterprise search show that a balance in the tradeoff leads to the best NDCG and good (predicted) clickthrough