Improving Evaluation of Machine Translation Quality Estimation

Quality estimation evaluation commonly takes the form of measurement of the error that exists between predictions and gold standard labels for a particular test set of translations. Issues can arise during com-parison of quality estimation prediction score distributions and gold label distribu-tions, however. In this paper, we provide an analysis of methods of comparison and identify areas of concern with respect to widely used measures, such as the ability to gain by prediction of aggregate statistics specific to gold label distributions or by optimally conservative variance in predic-tion score distributions. As an alternative, we propose the use of the unit-free Pear-son correlation, in addition to providing an appropriate method of significance testing improvements over a baseline. Compo-nents of WMT-13 and WMT-14 quality es-timation shared tasks are replicated to re-veal substantially increased conclusivity in system rankings, including identification of outright winners of tasks