Figure 1. Ask Multimodal Questions, Get Multimodal Answers

www.mitre.org/resources/centers/it This paper considers multimodal systems, resources, and evaluation. We first motivate the value of multimodal information access with a vision of multimodal question answering and an example of content based access to broadcast news video. We next describe intelligent multimodal interfaces, define terminology, and summarize a range of applications, required corpora, and associated media. We then introduce a jointly created roadmap for multimodality and show an example of an open source multimodal spoken dialogue toolkit. We next describe requirements for and an abstract architecture of multimodal systems. We conclude discussing multimodal collaboration, multimodal instrumentation, and multilevel evaluation. 1. Multimodal Question Answering A long range vision of ours is to create software that will support natural, multimodal information access. As implied by Figure 1, this suggests transforming the conventional information retrieval strategy of keyword-based document/web page retrieval into one in which multimodal questions spawn multimodal information discovery, multimodal extraction, and personalized multimodal presentation planning. In Figure 1 the user of the future is able to naturally employ a combination of spoken language, gesture, and perhaps even drawing or humming to articulate their information need which is satisfied using an appropriate coordinated integration of media and modalities, extracted from source media