Computer-aided analysis and design for spoken dialogue systems based on quantitative simulations

Abstract—Corpus-based analysis and design of spoken dialogue systems have been widely used. However, in such approaches the dialogue performance cannot be predicted before the system is on line, and the dialogue corpora need to be recollected if the system is modified or different conditions are assumed. Also, the effects of different factors, from the system’s dialogue strategies, speech recognition and understanding conditions and accuracy, to the user’s response pattern, etc., on the dialogue system performance cannot be quantitatively identified and analyzed, because they cannot be precisely controlled in different corpora. In this paper, a complete development of computer-aided anal-ysis and design approaches for spoken dialogue systems based on quantitative simulations is presented. With this approach the var-ious performance metrics of a dialogue system can be flexibly de-fined and numerically evaluated, such that the behavior and per-formance of the dialogue system can be well predicted and effi-ciently analyzed before the implementation of the real spoken dia-logue system is completed. How the different dialogue performance measures vary with respect to each of the many very complicated factors, regardless of whether it is caused by an individual compo-nent, by the overall system design, or by users ’ response patterns, can be separately identified, because all such factors can be pre-cisely controlled in the simulation. Several analysis examples are presented to show how the approach can be used, including selec-tion and tuning of speech understanding front end, system strategy design considering query factors and confirmation factors, and ob-jective estimates of user’s degree of satisfaction. This approach is therefore very useful for the analysis and design of spoken dialogue systems, although the online test, corpus-based analysis and user survey can always follow after the system is online. Index Terms—Computer simulation, dialogue strategy, dialogue system design, performance analysis, spoken dialogue. I