Automatic Evaluation of Machine Translation Quality

Any scientific endeavour must be evaluated in order to assess its correctness. In many applied sciences it is necessary to check that the theory adequately matches actual observations. In Machine Translation (MT), evaluation serves two purposes: relative evaluation allows us to check whether one MT technique is better than another, while absolute evaluation gives an absolute measure of performance, eg a score of 1 may mean a perfect translation. A peculiarity of MT evaluation is the fact that designing a proper evaluation measure is in itself a difficult problem. In many fields, there are obvious, measurable performance metrics, for example the difference between a prediction and an actual observed outcome. Because natural language is not exact and unambiguous, and also because its structure is complicated, estimating whether a translation is correct, or how far a translation is from correct is a lot more difficult. Two entirely different sequences of words (a.k.a. sentences) may be totally equivalent, while two sequences which differ in small detail can have entirely different meanings. Traditionally, the two main dimensions for evaluating translations are adequacy (does the translation convey the same meaning as the source text) and fluency (is the translation grammatically correct). Ideally, without any time or money constraints, MT output could be judged by human