Investigating Visuo-Haptic Interaction Fidelity and Display Metaphors on Near-Field Fine Motor Selection and Manipulation Tasks in Virtual Reality

Precision tasks are a very important part of virtual reality (VR). There are a wide variety of scenarios in virtual environments that employ these kinds of tasks, such as industrial machinery, electrical engineering, and robotic surgery. These precision tasks are much more effective when the apparatus is situated in a near-field space (at a range of about arm’s reach), so in order to optimize the experience in these tasks, it is important to maximize the realism of sensory fidelity. To this end, increasing sensory fidelity by adding cues such as haptic feedback and depth perception are key to regulating the perception-action cycle, which is the feedback loop in which users calibrate their actions to adapt to the sensory feedback they are given. In addition, the method that is used for tracking hand movements can play a very important role in determining how users interact with their environment. Hand movements in VR are not limited simply to the virtual hand matching the physical hand movements, as virtual movements can be scaled, offset, and redirected to fit the needs of the task design. Finally, precision tasks are mainly quantified by means of speed and accuracy, and those two cues are often in a trade-off. For this research, Fitts’ law is important in predicting this trade-off. To investigate these concepts, three experiments were conducted to determine how the perception-action cycle can be manipulated by changing sensory fidelity, interaction fidelity, and spatial distortions. The first experiment was an evaluation of user performance and calibration in a pick-and-place task in the presence or absence of haptic feedback and stereoscopic rendering. This experiment used a stereoscopic 3D TV as an immersive head-tracked display, and it used the PHANToM OMNI haptic device to create a sensation of force feedback. This experiment was designed to determine if haptic feedback, stereoscopic rendering, and spatial distortion affected user performance by the metrics of speed, accuracy, and economy of movement. The findings of this experiment revealed that haptic feedback and stereoscopic viewing improved this type of performance, and spatial distortions hindered it as well. In addition, as the level of sensory fidelity and spatial distortions became less immersive, users were required to make larger calibrations, which could be problematic in situations where quick calibrations are important. The second experiment was an evaluation of the speed-accuracy trade-off through the lens of Fitts\u27 law. The study that was originally used as a basis for Fitts\u27 law investigated performance in both selection and manipulation tasks, but only selection tasks are widely studied in the relevant literature, with manipulation tasks only recently being the subject of focus. The second study used a ring transfer task similar to the disc transfer task from Fitts\u27 original study to determine how pick-and-place tasks comply with the predictive model of Fitts\u27 law. In addition, this study aimed to determine the effects of haptic feedback, depth perception, and spatial distortion on task throughput as defined by Fitts\u27 law. The findings from this study revealed that haptic feedback and visuo-proprioceptive mismatch affected movement time and index of performance. Finally, the correlation between difficulty and time was moderately strong, but there are more factors to consider when applying Fitts\u27 law to manipulation tasks. The final experiment evaluated the effects of screen parallax and vergence-accommodation conflicts on speed, accuracy, and economy of movement in a pick-and-place task. Specifically, this research aimed to determine if user performance was stronger in positive screen parallax (in front of the screen) or negative screen parallax (behind the screen). In negative screen parallax space, the display can render more levels of depth more finely than in positive screen parallax space. In addition, vergence-accommodation conflicts can be different in positive screen parallax space than in negative screen parallax space specifically because the convergence point is behind the focus point in positive parallax while it is behind the focus point in negative parallax. The results of the study showed that user performance was stronger in negative parallax, likely due to improved distance judgments. This held true both when moving between parallax zones and within the same parallax zone