Benefits and Challenges of Real-Time Uncertainty Detection and Adaptation in a Spoken Dialogue Computer Tutor

We evaluate the performance of a spoken dialogue system that provides substantive dynamic responses to automatically detected user affective states. We then present a detailed system error analysis that reveals challenges for real-time affect detection and adaptation. This research is situated in the tutoring domain, where the user is a student and the spoken dialogue system is a tutor. Our adaptive system detects uncertainty in each student turn via a model that combines a machine learning approach with hedging phrase heuristics; the learned model uses acoustic-prosodic and lexical features ex-tracted from the speech signal, as well as dialogue features. The adaptive system varies its content based on the automatic uncertainty and correct-ness labels for each turn. Our controlled experimental evaluation shows that the adaptive system yields higher global performance than two non-adaptive control systems, but the difference is only significant for a subset of students. Our system error analysis indicates that noisy affect labeling is a major per-formance bottleneck, yielding fewer than expected adaptations thus lower than expected performance. However, the percentage of received adapta-tion correlates with higher performance over all students. Moreover, when uncertainty is accurately recognized and adapted to, local performance is significantly improved. Keywords: fully automated spoken dialogue tutoring system, automatic affect detection and adaptation, accompanying manual annotation, controlled experimental evaluation, error analysis Preprint submitted to Speech Communication Special Issue: Sensing Emotion and AffectJanuary 19, 2011 1