Linguistic-based evaluation criteria to identify statistical machine translation errors

Comunicació presentada a: the 14th annual Conference of the European Association for Machine Translation,, celebrada del 27 al 28 de maig de 2010 a Saint-Raphaël, França.Machine translation evaluation methods are highly necessary in order to analyze the performance of translation systems. Up to now, the most traditional methods are the use of automatic measures such as BLEU or the quality perception performed by native human evaluations. In order to complement these traditional procedures, the current paper presents a new human evaluation based on the expert knowledge about the errors encountered at several linguistic levels: orthographic, morphological, lexical, semantic and syntactic. The results obtained in these experiments show that some linguistic errors could have more influence than other at the time of performing a perceptual evaluation.The N-II machine translation system developed at the UPC has been funded by the European Union under the integrated project TC-STAR: Technology and Corpora for Speech to Speech Translation (IST-2002-FP6-506738), the Spanish Government under the BUCEADOR project (TEC2009-14094-C04-01), and partially funded by the Spanish Department of Education and Science through the Juan de la Cierva fellowship program