Testing Driver Attention in Virtual Environments Through Audio Cues

Virtual Reality has risen in popularity this decade due to the high level of interactive experience it provides to its users by encapsulating them in various computer-generated environments. These can range from video-games to real-life simulations where they try to create a near 3D experience similar to that one would have in the real world. The introduction of audio on video-games or simulations can change not only the experience and feel of the environment but also improve the perception of their surroundings, including new elements otherwise not perceivable without audio. Human’s audio attention process is complex and can change depending on the current context, time and depend on each personality, making it interesting to be studied. This dissertation’s main goal is to improve the agent’s behaviour by creating an audio attention model to allow them to be aware of audio and react to the scenario’s soundscape and to a specific event. By improving the agent’s perceptions and actions, this makes their decision making better fit the virtual environment they are inserted. However, this requires studying the current scenario they belong to as well as the role they represent. To create such attention model calls for the existence of audio on the current scenario. This led to the study of a city soundscape and its elements to comprehend what audio events exist and how relevant they are in that context. Then, it was necessary to fit this selection to the existent scenario and/or create new ways to use them without losing the realism level. Also, conventional audio output decoders do not fully represent on recreating the human auditory system due to its unique characteristics. 3D Audio and Binaural systems were introduced on the virtual space to better replicate the expected experience whilst immersed. Afterwards, it was evaluated different ways on how to test audio attention on virtual environments and required studying research methods to do so. Human-Computer Interaction instructed on how to make applications more user-friendly and intuitive with the introduction of new modalities. Within-Subject design and Grounded theory proved to be useful in gathering data to use for the attention model, but this data provided limited usefulness due to the low number of participants and inconclusive in a lot of aspects, limiting the agent’s audio attention model performance. Lastly, there was implemented an approach to automatic audio classification by analysing the output signal mix and then with the results obtained, control the agent’s perception by correctly or incorrectly identify the audio source and determine what the agent should do if the case. The usage of a Convolutional Neural Network with an embedded attention model was proposed to analyse the signal by evaluating it on a fully trained network, thanks to Google’s audio dataset. Results were technically correct but were ambiguous on certain aspects and required some analysis to conclude what elements are present. This was also not viable due to how processing demanding it is and to other software, constraints made obtaining results in real-time impossible. Explanations as to why this and other limitations happened and solutions were introduced to solve most problems on future interventions for related types of works. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement N. 723386 (SIMUSAFE)