Evaluating Algorithms for the Generation of Referring Expressions: Going beyond Toy Domains.

We describe a corpus-based evaluation method-ology, applied to a number of classic algorithmsin the generation of referring expressions. Fol-lowing up on earlier work involving very simpledomains, this paper deals with the issues asso-ciated with domains that contain ‘real-life’ ob-jects of some complexity. Results indicate thatstate of the art algorithms perform very differ-ently when applied to a complex domain. More-over, if a version of the Incremental Algorithm isused then it becomes of huge importance to se-lect a good preference order, because some pref-erence orders are prone to generating very un-natural output. Finding good preference orders,however, can be difficult, as we show. These re-sults should contribute to a growing debate onthe evaluation of nlg systems, arguing in favourof carefully constructed balanced and semanti-cally transparent corpora