Explorations in a Natural Language Multimodal Information Access Environment

This paper addresses the problem of communication with a multimodal system, where communication integrates things other than words, and try to give a unified and concrete, albeit simplified and limited, working solution for information access dialogues. 1 Introduction In multimodal interaction systems [Maybury,1993a] interactivity can mean something that goes beyond what a person is used to in nature: on one side the possibility for the computer of using different communication media, such as graphics, images, written or synthetized language etc. in a fast and coordinated way extends conceptually the bandwidth of communication toward the person. On the other side the possibility for the person of directly manipulating entities in combination with language opens new possibilities that change the meaning of utterance and exchange. We address the problem of communication with a multimodal system, where communication integrates things other than words, and try to give a unified and concre..