Automatic Machine Translation Evaluation using a Source and Reference Sentence with a Cross-lingual Language Model

As the performance of machine translation has improved, the need for a human-like automatic evaluation metric has been increasing. The use of multiple reference translations against a system translation (a hypothesis) has been adopted as a strategy to improve the performance of such evaluation metrics. However, preparing multiple references is highly expensive and impractical. In this study, we propose an automatic evaluation method for machine translation that uses source sentences as additional pseudo-references. The proposed method evaluates a translation hypothesis via regression to assign a real-valued score. The model takes the paired source, reference, and hypothesis sentences together as input. A pre-trained large-scale cross-lingual language model encodes the input to sentence vectors, with which the model predicts a human evaluation score. The results of experiments show that our proposed method exhibited stably higher correlation with human judgements than baseline methods that solely depend on hypothesis and reference sentences, especially when the hypotheses were very high- or low-quality translations