Dealing with Ties in Rank Correlation

In the field of Information Retrieval (IR), rankings of systems evaluated under different conditions are often compared to each other. This measure of correspondence between rankings, termed as rank correlation, must accurately capture the scenario for which the correlation is computed. Very often, these rankings may have tied systems, for which new correlation coefficients arise. It is important that these coefficients account for the new scenarios in the presence of ties. It is also important that these coefficients provide some flexibility to the person performing the correlation to introduce artificial ties when items are so close to each other that, for practical purposes, they may be considered as tied. Accounting for these different scenarios of tied items in rankings permits performing per topic comparisons of IR systems, which was earlier limited due to the presence of ties on individual topics. Comparing rankings at the topic level, the expected variability of the rankings can be studied to potentially improve the systems on the topics for which they perform poorer than average. We show the application of these new correlation coefficients with two typical IR experiments.Computer Science | Multimedia Computin