Development of a Virtual Reality application focused on rehabilitation therapies combining VR glasses and Deep Learning

In this work a Virtual Reality application is developed and focused on the physical rehabil-itation of people with injuries or neurodegenerative diseases, controlled by the movement of the body registered by a camera and through the helmet and controls of the virtual reality set. The game proposes a simple task of arranging everyday objects found in the virtual world. The application integrates 6 deep learning models, trained with a tailored dataset, to predict effort values of the user playing the game. The proposed game involves exercises with simple movements that do not require much strength, elasticity or energy, so they are especially suitable for disorders such as Parkinson’s disease, and mental exercises related to context, orientation and simple logic, especially important in cases of other dementia diseases such as Alzheimer’s disease, as for many other injuries or disorders. The application requires the enabling of some physical elements: • A camera captures video of the person using the application, and performs by means of aritificial intelligence models in Python the identification of joint movements, with the help of one or more models trained for this task. • The virtual environment is developed in the Unity engine, in which the patient will interact with the virtual elements using the controls and the helmet. Two base stations cover the play area and pick up the signal from the controls and helmet according to the user’s movements. • In response to the received movements and actions, the virtual reality application will update the scene the user sees on the HTC Vive goggles so that the user can control his movement in a virtual room and perform a series of simple actions to complete the game objective. This implementation is done in the Unity 3D development environment, driven through Steam VR for the HTC Vive headset. Finally, the application will provide a performance report with some metrics useful for the evaluation. It is desired to look at two practical use cases, one for rehabilitation in a clinical setting (hospital or physiotherapy), and one for rehabilitation of people with Parkinson’s or Alzheimer’s disease