D4.4 Report on Summative Usability and User Experience Evaluation of Advanced ARETE Prototypes

This is the last deliverable of WP4, which is built upon the existing work of the HCI team throughout the ARETE project with the goal of enhancing the usability and user experience of the prototypes of the four Pilot studies. Formative evaluation studies have been documented in previous WP4 deliverables. In this D4.4, summative evaluation studies, which have been conducted with representative users, namely, teachers with heterogeneous backgrounds, during the final year of the project. The processes and results of the summative evaluation studies for Pilot 1 and Pilot 2, which understandably focus on the pedagogical effectiveness of the interventions with the AR apps developed, are reported in detail in D6.5 (M42). From the HCI perspective, we present the key findings based on the data of the adapted Technology Acceptance Model (TAM) for Pilot 2 (NB: the small sample size of Pilot 1 did not allow any meaningful analysis). Both the mean ratings for Perceived Usefulness and Behavioural Intention were higher than 6.0 (out of 7.0), suggesting that the teachers were convinced about the strong potential of the AR apps for educational uses and exhibited high intention to adopt them. The mean rating for Perceived Ease of Use was high with 5.7, though there is room for improvement. For Pilot 3 on the PBIS-AR app, the summative evaluation was based on a survey, which was translated into Italian and Dutch to be completed by the teachers participating in the workshop in the respective venue. However, at the time of writing up this deliverable, the data are yet to be collected in the Netherlands. Results presented here are based on 30 participants in Italy, who were asked to try out the MirageXR-based PBIS-AR app in its Modelling, Teach, Discovery and Practice mode. Overall, the participants had a high level of acceptance of the PBIS-AR app. They appreciated the power of AR that allows different real- life situations to be simulated in the virtual worlds, demonstrating behaviours on which students are enabled to reflect. In addition, most participants seemed convinced that students would find the app enjoyable and motivating, especially the likeable main virtual character Arpro. Nonetheless, as a new technology, there is still room to improve, including the ease of manipulating the characters from the technical perspective and the clarity of answer options from the pedagogical perspective. For Pilot 4, a workshop was conducted to usability test the recent version of MirageXR (v.1.9). It involved 29 teachers coming from 9 countries. Results of observations and survey suggested that the teachers had a moderate level of acceptance of it. The main positive outcomes are that the teachers expressed a high level of enthusiasm about the strong potential of MirageXR for a variety of educational purposes and that the learnability of the app was high. Many of the participating teachers needed some kinds of support to overcome the initial barriers (e.g., registration, calibration). Nonetheless, as an emerging technology, inevitably there is still room to improve. We identified 24 usability problems (UPs), including app freezes and crashes, editing objects, and imprecision. The UPs can inform the ongoing work of MirageXR. Finally, the HCI team were inspired to design, develop and evaluate an innovate AR- based educational tool: Tutor In-sight - a mixed reality avatar augmented into the student’s workspace with a teacher’s dashboard. The related research effort resulted in the paper accepted in the prestigious Conference on Computing Systems in Human Factors (CHI’23).This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 856533