A Methodology for Evaluating Spoken Language Dialogue Systems and Their Components

As spoken language dialogue systems (SLDSs) proliferate in the market place, the issue of SLDS evaluation has come to attract wide interest from research and industry alike. Yet it is only recently that spoken dialogue engineering researchers have come to face SLDSs evaluation in its full complexity. This paper presents results of the European DISC project concerning technical evaluation and usability evaluation of SLDSs and their components. The paper presents a methodology for complete and correct evaluation of SLDSs and components together with a generic evaluation template for describing the evaluation criteria needed. 1