Modelling Score Distributions Without Actual Scores

Score-distribution models are used for various practical pur-poses in search, for example for results merging and thresh-old setting. In this paper, the basic ideas of the score-distributional approach to viewing and analysing the ef-fectiveness of search systems are re-examined. All recent score-distribution modelling work depends on the availabil-ity of actual scores generated by systems, and makes as-sumptions about these scores. Such work is therefore not applicable to systems which do not generate or reveal such scores, or whose scoring/ranking approach violates the as-sumptions. We demonstrate that it is possible to apply at least some score-distributional ideas without access to real scores, knowing only the rankings produced (together with a single effectiveness metric based on relevance judgements). This new basic insight is illustrated by means of simulation experiments, on a range of TREC runs, some of whose re-ported scores are clearly unsuitable for existing methods