Design and Evaluation of Metaphor Processing Systems

System design and evaluation methodologies receive significant attention in natural language processing (NLP), with the systems typically being evaluated on a common task and against shared datasets. This enables direct system comparison and facilitates progress in the field. However, computational work on metaphor is considerably more fragmented than similar re-search efforts in other areas of NLP and semantics. Recent years have seen a growing interest in computational modelling of metaphor, with many new statistical techniques opening routes for improving system accuracy and robustness. However, the lack of a common task definition, shared dataset and evaluation strategy makes the methods hard to compare, and thus hampers our progress as a community in this area. The goal of this article is to review the system features and evaluation strategies that have been proposed for the metaphor processing task, and analyse their benefits and downsides, with the aim of identifying the desired properties of metaphor processing systems and a set of requirements to their evaluation. 1