DQR Test Suites for a Qualitative Evaluation of Spoken Dialog Systems : from Speech Understanding to Dialog Strategy

International audienceGenerally speaking, the evaluation of spoken dialog systems is based on objective metrics that are supposed to offer a complete survey of the system's behaviour. Unfortunately, this approach boils down to a measurement of the overall performances of the system. Despite its indisputable interest, this quantitative approach lacks some predictive power to enable a really informative evaluation. On the contrary, we propose a complementary methodology that intends to respect this criterion of predictability. Inspired by some NLP works (Rolbert and Sabatier 1996), it is based on the definition of DQR tests which are specific to each linguistic phenomenon, what warrants the qualitative and predictive nature of the evaluation. This paper describes the adaptation of the DQR methodology to speech dialog systems. We have thus defined a multi-level framework of evaluation that concerns speech understanding as well as the dialog strategy. This paper focuses mainly on speech understanding. At first, the DQR methodology is reviewed. Then, numerous examples illustrate the practical elaboration of DQR test suites. Finally, we highlight the achievement of an evaluation session on several kinds of real spoken dialog systems