Full-Body Motion Tracking In Immersive Virtual Reality - Full-Body Motion Reconstruction and Recognition for Immersive Multiplayer Serious Games

The release of consumer-grade virtual reality head-mounted displays contributed to the development of immersive applications that convey an illusion of being present in the virtual environment. This great potential of virtual reality is promising not only for the entertainment industry but also for education and health. However, the head-mounted display obstructs the players' view of the real environment, causing them to see neither the real environment nor their bodies or those of their teammates and opponents. Therefore, full-body motion reconstruction is essential to improve the sense of presence and interaction among users. Nevertheless, due to the lack of users' motion data, many popular virtual reality games focus solely on upper-body movements and show only controllers or floating hands. Moreover, full-body motion recognition is crucial to ensure that users perform desired physical activities correctly, either to improve health outcomes or to lower the risk of injury. The contributions in this thesis include the reconstruction and recognition of full-body movements using off-the-shelf virtual reality devices. However, such a motion tracking system requires many sensors to be attached to the body, making it difficult to set up and uncomfortable to wear. Therefore, as the first contribution, the number of sensors is reduced to not restrict the user's movements. A reduction in sensors is also required in health-based applications as patients with physical limitations often cannot hold or wear additional devices. To this end, inverse kinematics methods are explored and their parameters are optimized to estimate the full-body pose with high accuracy and low latency. Because high latency between the user's movements and the corresponding visual feedback on the head-mounted display causes cybersickness, the effect of increased end-to-end latency on user experience and performance is investigated as the second contribution. Here, an end-to-end latency threshold that elicits significant cybersickness and causes users to need significantly more time to complete a task is identified. As the third contribution, machine learning algorithms are employed to identify suitable sensor positions for reliable full-body motion recognition. Thereby, the entire movement is analyzed and potential activity execution errors are identified. The elaborated model on full-body motion reconstruction and recognition is prototypically implemented and validated in the context of two serious games: (1) an exergame designed to motivate players to train specific movements and (2) a multiplayer training simulation for police forces to enable training of stressful situations. In the exergame, the system's capability has been demonstrated to recognize the activity execution errors and provide appropriate feedback so that players can improve their movements. By means of the training simulation, statistical significance and effect sizes have been analyzed to examine the impact of full-body avatars in contrast to an abstract representation with head and hands on stress level. Thereby, an empirical study with police forces showed the added value of full-body avatars, which improve the feeling of presence and enable communication via body language and gestures