Computer vision, eyetracking, spoken dialog systems, and evaluation: Challenges and opportunities

Spoken dialog systems have matured to the point where the underlying technologies of speech recognition, natural language understanding, dialog management, natural language generation, and speech synthesis are available for many languages and are portable to various domains. However, when people communicate, spoken language is accompanied by a host of other inputs and outputs. The most immediately salient of these processes is input and output perceived visually: gestures, eye gaze, posture, and the like. We focus on three kinds of visual input: the first can be roughly called visual awareness, and includes factors such as what objects (including people) are in the visual scene, how far away they are from the computer, and so forth. Certain types of visual input such as posture, head pose, blink rate, and pupil dilation are more fine-grained and have to do with analyzing an individual's cognitive and affective states; we'll call this cluster personal perception. The third is eyetracking: where the user is looking. Here we spell out a number of challenges posed by the introduction of the visual channel into spoken dialog systems, and enumerate some opportunities that vissual input provides. We focus especially on how these challenges relate to evaluation – since evaluation of a spoken dialog system’s behavior is a central aspect of dialog system research and is especially critical for important applications such as intelligent tutoring systems