Target suites for evaluating the coverage of text generators

Our goal is to evaluate the grammatical coverage of the surface realization component of a natural language generation system by means of target suites. We consider the utility of re-using for this purpose test suites designed to assess the coverage of natural language analysis / understanding systems. We find that they are of some interest, in helping inter-system comparisons and in providing an essential link to annotated corpora. But they have limitations. First, they contain a high proportion of ill-formed items which are inappropriate as targets for generation. Second, they omit phenomena such as discourse markers which are key issues in text production. We illustrate a partial remedy for this situation in the form of a text generator that annotates its own output to an externally specified standard, the TSNLP scheme. 1. Test suites and target suites The evaluation of natural language generation (NLG) systems is an issue that few authors have addressed seri-ously to date. Among the principal reasons is the difficulty of defining what the input should be and, indeed, of assess-ing the quality of the output (cf., (Sparck-Jones and Gal-liers, 1996)). Evaluations conducted to assess the adequac