ccb cs jhu edu

j schroeder ed ac uk This paper analyzes the translation qual-ity of machine translation systems for 10 language pairs translating between Czech, English, French, German, Hungarian, and Spanish. We report the translation quality of over 30 diverse translation systems based on a large-scale manual evaluation involv-ing hundreds of hours of effort. We use the human judgments of the systems to analyze automatic evaluation metrics for translation quality, and we report the strength of the cor-relation with human judgments at both the system-level and at the sentence-level. We validate our manual evaluation methodol-ogy by measuring intra- and inter-annotator agreement, and collecting timing informa-tion