On Choosing an Effective Automatic Evaluation Metric for Microblog

Popular microblogging services, such as Twitter, are engag-ing millions of users who constantly post and share infor-mation about news and current events each day, resulting in millions of messages discussing what is happening in the world. To help users obtain an overview of microblog con-tent relating to topics and events that they are interested in, classical summarisation techniques from the newswire domain have been successfully applied and extended for use on microblogs. However, much of the current literature on microblog summarisation assumes that the summarisation evaluation measures that have been shown to be effective on newswire, are still appropriate for evaluating microblog summarisation. Hence, in this paper, we aim to determine whether the traditional automatic newswire summarisation evaluation metrics generalise to the task of microblog sum-marisation. In particular, using three microblog summarisa-tion datasets, we determine a ranking of summarisation sys-tems under three automatic summarisation evaluation met-rics from the literature. We then compare and contrast this ranking of systems produced under each metric to system rankings produced through a qualitative user evaluation, with the aim of determining which metric best simulates hu-man summarisation preferences. Our results indicate that, for the automatic evaluation metrics we investigate, they do not always concur with each other. Further, we find that Fraction of Topic Words better agrees with what users tell us about the quality and effectiveness of microblog sum-maries than the ROUGE-1 measure that is most commonly reported in the literature