A Mixed Reality System for Upper Limb Prosthesis Training & Control

Upper limb prosthesis training has traditionally been limited to physical setups within clinical settings. In this work, we present an augmented reality (AR) system for prosthesis control and training. Holographic myoelectric prosthesis based on the virtual model of the Modular Prosthetic Limb (vMPL) was integrated into the system. The AR version of an existing physical prosthesis training setup, the Prosthetic Hand Assessment Measure (PHAM), was developed. The AR version of the PHAM — HoloPHAM, offers amputees the advantage to train with pattern recognition, at their own time and convenience, pre- and post-prosthesis fitting. To enrich the experience of interaction with holograms, biofeedback was provided to the users. The HoloPHAM incorporates both visual (VF) and vibrotactile feedback (TF) for intuitive control and object manipulation. We provide rigorous analysis and testing of our system, focusing on its ability to simulate reach, grasp, and touch in AR. Kinematic aspects of rotation, translation, joint dynamics, and trajectory, between PHAM and HoloPHAM, are compared under the two conditions: a) with only VF and b) with both VF and TF. For clinical considerations, kinematic variability, range of motion (ROM) and task completion times, between the two modalities were also compared. For able-bodied subjects (N=5), rotational and translational errors between the physical and AR space were -1.11deg and 0.38cm, respectively. Results suggest that the condition with both VF and TF reduce end-point variability; ultimately, improving the decision-making ability to grasp objects in AR space faster. Kinematic joint dynamics, variability, and ROM between the PHAM and HoloPHAM showed statistically significant equivalence with the task completion times linearly correlated (R2 = 0.7681) between the two modalities. A multiple day pilot study involving subjects (N=3 able-boded) training exclusively with the HoloPHAM showed statistically significant (p