Better Summarization Evaluation with Word Embeddings for ROUGE

ROUGE is a widely adopted, automatic evaluation measure for text summariza-tion. While it has been shown to corre-late well with human judgements, it is bi-ased towards surface lexical similarities. This makes it unsuitable for the evalua-tion of abstractive summarization, or sum-maries with substantial paraphrasing. We study the effectiveness of word embed-dings to overcome this disadvantage of ROUGE. Specifically, instead of measur-ing lexical overlaps, word embeddings are used to compute the semantic similarity of the words used in summaries instead. Our experimental results show that our pro-posal is able to achieve better correlations with human judgements when measured with the Spearman and Kendall rank co-efficients.