D6.5 Pilots Performance Analysis

This document reports on the evaluation of ARETE pilots 1, 2 and 3 . It presents the data collected in these three pilots in the ARETE project and summarizes and discusses results and outcomes. Overall, the evaluation was designed to identify effects of the educational use of AR-enhanced applications on student learning and behaviour and on teaching and learning processes in a variety of educational settings. ARETE pilot 1 focused on AR-enhanced literacy attainment. This study included a pre and post test design with intervention and control groups. 29 students and 6 teachers completed the evaluation in this study which comprised standardized student reading and spelling tests pre and post intervention and surveys and interviews with teachers before and after the intervention. For pilot 1, central outcomes of the ARETE evaluation show that students were able to significantly improve their reading and spelling skills using the AR-enhanced literacy learning app. Students and teachers overall perceived the app as motivating and useful, even though potential for improvement was identified, e.g., with regards to technology and bug fixing practices. Pilot 2 was about AR-enhanced STEM learning in the fields of Mathematics and Science; two respective apps were tested and evaluated in a study that involved 1988 students and 91 teachers from 11 countries. The study design, which included an intervention and control group approach, was based on standardized knowledge tests for students administered pre and post intervention and a third time six weeks later to measure knowledge retention. Teachers filled in surveys and participated in focus groups before and after the intervention. Results from pilot 2 indicate that the AR-enhanced applications successfully supported student STEM learning. Students’ knowledge test results improved significantly after using the app, and the improvement was significantly higher compared to the control group. Teachers found various ways to integrate the app into their teaching and learning processes and confirmed positive effects on student motivation and classroom engagement. Problems described referred, e.g., to technology and teachers and students involved. Pilot 3 was interested in the evaluation of the impact of an AR-enhanced PBIS intervention on students’ development of socio-emotional skills and learning of behavioural expectations. A pre-test/post-test between group design was adopted, and data were collected on a sample of around 300 Italian and Dutch students over the course of two consecutive school years. Students were asked to complete a set of questionnaires assessing their acquisition of social, emotional, and cognitive skills and their experience with the AR mobile application. Teachers rated students’ acquisition of the behavioural expectations taught during the intervention and their own experience with the application. Overall, students assigned to both the educational conditions (with and without the support of AR) performed better on all the behavioural expectations; moreover, students participating in the augmented intervention rated their experience with the AR application quite satisfactory. Overall, the data presented in this deliverable illustrate that the educational use of AR- enhanced applications can have positive effects in teaching and learning processes, such as increased student motivation and classroom engagement, successful student learning and improved student behaviour. According to the teachers involved, the AR-enhanced interventions in ARETE tended to be successful in supporting student learning processes and integrated rather well into teaching and learning processes. On the other hand, the data also reveal potential barriers and problems of the educational use of AR. In this context, a significant share of such problems identified in the studies refer to technical problems with regards to the applications and devices used; further barriers refer to pedagogical aspects, e.g., in relation to the time needed and the adaption of teaching and learning processes necessary to integrate innovative AR-enhanced methods and approaches. The challenges identified can be addressed in future studies to facilitate the further uptake of AR technologies in educational contexts which, according to the data collected, offers promising potential for effectively supporting teaching and learning processes.This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 856533