The role of evaluation in bringing NLP to AAC: A case to consider

Evaluation of prototype AAC technologies is a very difficult task for several reasons. Among these are the difficulties inherent in evaluating a ``partial' ' system-- i.e., one whose focus is on a single aspect of an overall system. For example, for several years, we have been applying natural language processing techniques to the field of AAC in order to develop intelligent communication aids that attempt to provide linguistically ``correct' ' output while increasing communication rate. Our focus has been on the processing and system knowledge required in order to expand the user's input. The outcome motivating this project was primarily rate enhancement. While a research prototype was developed at the University of Delaware based on an NLP technique we called COMPANSION (because it takes a COMPressed message and through expANSION, converts it into a well-formed sentence), its practical deployment and outcome evaluation faces several difficulties. These are primarily because the focus of the technique was on processing, but an evaluation requires, and is dependent on, an entire device (i.e., input interface, processing, and output interface). We include an informal experiment which allows a partial analysis of the technique. While such experiments are unable to shed light on possible outcomes of system use, they do validate some assumptions and point out differences among users from different populations. In continuing our investigation of how COMPANSION might be incorporated into a viable AAC device, a joint project between the University of Delaware and the Prentke Romich Company was undertaken to investigate the possibility of incorporating COMPANSION into a viable communication device for a particular population. The development methodology for this project includes ongoing evaluation of sub-components of the system and tailoring of the system processing to the specific population through a data collection and analysis effort. A portion of the collected data has been set aside for testing purposes. 1