Modality tracking in the multimodal Bell Labs Communicator

In this paper, we describe our efforts in designing and build-ing the multimodal Bell Labs Communicator system. Sev-eral innovations are necessary for generalizing the speech-only user interface and the semantic/pragmatic algorithms to handle both speech and visual modalities, including changes to the parser, semantic and pragmatic modules to better integrate the user input from different modalities within and across dialogue turns, as well as extending the concept of e-forms to include the visual modality. The sec-ond part of the paper describes the design of a natural, con-sistent and efficient multimodal user interface. We intro-duce the concept of the “dominant modality”: the system adaptively tracks the most probable interaction mode and suggests that modality to the user. Modality tracking pro-vides an elegant solution to the “turn-taking ” issue in mul-timodal spoken dialogue systems. 1