*Sail Labs GmbH

In this paper we present a corpus-based method to evaluate the translation quality of machine translation (MT) systems. We start with a shallow analysis of a large corpus and gradually focus the attention on the translation problems. The method constitutes an efficient way to identify the most important grammatical and lexical weaknesses of an MT system and to guide development towards improved translation quality. The evaluation described in the paper was carried out as a cooperation between an MT technology developer, Sail Labs, and the Computational Linguistics group at the University of Zürich. 1. Different types of evaluation for different purposes Sail Labs does various types of translation quality (TQ) evaluations (absolute, comparative, text and sentence-based) and uses different methods (glass-box, black-box evaluations, pre- and postrelease, using linguistic test suites and real text corpora). Most of the evaluations are from a developer's rather than a user’s point of view. Please note that these evaluations wer