How do Humans Evaluate Machine Translation

In this paper, we take a closer look at the MT evaluation process from a glass-box perspective using eye-tracking. We an-alyze two aspects of the evaluation task – the background of evaluators (monolin-gual or bilingual) and the sources of in-formation available, and we evaluate them using time and consistency as criteria. Our findings show that monolinguals are slower but more consistent than bilinguals, especially when only target language in-formation is available. When exposed to various sources of information, evaluators in general take more time and in the case of monolinguals, there is a drop in consis-tency. Our findings suggest that to have consistent and cost effective MT evalua-tions, it is better to use monolinguals with only target language information.