Speech and Gesture Multimodal Control of a Whole Earth 3D Visualization Environment

Abstract | A growing body of research shows several advantages to multimodal interfaces in-cluding increased expressiveness, flexibility, and user freedom. This paper investigates the de-sign of such an interface that integrates speech and hand gestures. The interface has the addi-tional property of operating relative to the user and can be used while the user is in motion or stands at a distance from the computer display. The paper then describes an implementation of the multimodal interface for a whole earth 3D visualization environment which presents navi-gation interface challenges due to the large mag-nitude of scale and extended spaces that is avail-able. The characteristics of the multimodal in-terface are examined, such as speed, recogniz-ability of gestures, ease and accuracy of use, and learnability under likely conditions of use. This implementation shows that such a multimodal interface can be eective in a real environment and sets some parameters for the design and use of such interfaces