Finite-state multimodal parsing and understanding

Multimodal interfaces require effective parsing and understanding of utterances whose content is dis-tributed across multiple input modes. Johnston 1998 presents an approach in which strategies for mul-timodal integration are stated declaratively using a unification-based grammar that is used by a multi-dimensional chart parser to compose inputs. This approach is highly expressive and supports a broad class of interfaces, but offers only limited potential for mutual compensation among the input modes, is subject to significant concerns in terms of computa-tional complexity, and complicates selection among alternative multimodal interpretations of the input. In this paper, we present an alternative approach in which multimodal parsing and understanding are achieved using a weighted finite-state device which takes speech and gesture streams as inputs and out-puts their joint interpretation. This approach is sig-nificantly more efficient, enables tight-coupling of multimodal understanding with speech recognition, and provides a general probabilistic framework for multimodal ambiguity resolution.