SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT)

In this shared task, we present evaluations on two related tasks Paraphrase Identification (PI) and Semantic Textual Similarity (SS) sys-tems for the Twitter data. Given a pair of sentences, participants are asked to produce a binary yes/no judgement or a graded score to measure their semantic equivalence. The task features a newly constructed Twitter Para-phrase Corpus that contains 18,762 sentence pairs. A total of 19 teams participated, sub-mitting 36 runs to the PI task and 26 runs to the SS task. The evaluation shows encourag-ing results and open challenges for future re-search. The best systems scored a F1-measure of 0.674 for the PI task and a Pearson corre-lation of 0.619 for the SS task respectively, comparing to a strong baseline using logis-tic regression model of 0.589 F1 and 0.511 Pearson; while the best SS systems can of-ten reach>0.80 Pearson on well-formed text. This shared task also provides insights into the relation between the PI and SS tasks and sug-gests the importance to bringing these two re-search areas together. We make all the data, baseline systems and evaluation scripts pub-licly available.1