Improving interactive multisensory simulation and rendering through focus on perceptual processes

In this dissertation we present a novel approach to the design of multisensory interactive applications. We develop methods that improve the effectiveness of this type of application by incorporating the existing understanding of the human perceptual system. We quantitatively demonstrate the validity of our techniques through user studies and laboratory measurement. Explicitly including the human user as a part of the model for interactive application design, we propose a design approach for training simulators that augments the dynamical simulation of an interactive task (e.g., a surgical procedure) with feedback that highlights the aspects of the interaction that are perceptually pertinent for the purposes of training. We show how this type of augmentation can improve the training effectiveness of a simulator without necessitating more expensive rendering hardware. To make our perceptually-based augmentation technique more useable by application designers, we propose a decomposition approach to simplify the general process of developing the appropriate augmentation for a training simulator. We validated our approach by applying it to the design of a training simulator for a haptic search task; we conducted a user study that found a statistically significant improvement in the training effectiveness of the augmented simulator vs. an unaugmented simulator. We propose specific guide-lines for how existing psychophysical experimental results can be used to build augmentations for training simulators. We also developed a novel rendering architecture for distributed interactive applications that is suitable for the type of perceptually augmented simulator described above. Our architecture uses prediction of perceptually pertinent interaction events to achieve application latency and asynchrony bounds that can be constrained to within psychophysically established thresholds.Ph.D.Includes bibliographical references (p. 123-130)by Timothy Edmund