The realist evaluation of educational technology

PURPOSE. This thesis considers the best way to address the challenges faced by educators, institutions and funding bodies trying to not only develop and implement educational technology successfully but tackle the challenge of understanding and evidencing what works (and what does not) and why. The aim of the research was to find and validate an evaluation method that provided usable and useful evidence. APPROACH. A range of evaluations were undertaken to elicit the strengths and weaknesses of different approaches, augmented by drawing upon the experiences and outcomes published by others. An analysis of the issues was made and significance of the problem established. The problem being premature timing, unsuitable models, rapid change, complex implementation chains, inconsistent terminology, ideology and marketisation. A tailored realist evaluation framework was proposed as an alternative method and it was tested to evaluate an institutional lecture capture (LC) initiative. FINDINGS. The theory-driven realist approach provided a level of abstraction that helped gather evidence about wider influences and theories of potential future impact of the LC programme and its linked policy. It proved valuable in generating real and practical recommendations for the institution, including what more could be done to improve uptake and support embedding in teaching and learning, from practice, policy and technological points of view. It identified some unanticipated disadvantages of LC as well determining how and when it was most effective. PRACTICAL IMPLICATIONS. A Realist Evaluation of Technology Initiative (RETI) framework has been produced as tool to aid the rapid adoption of the approach. Recommendations for future research and seven guiding principles have been proposed to encourage the formation of a community of realist evaluative researchers in educational technology. ORIGINALITY/VALUE. The rigorous application of a tailored realist evaluation framework (RETI) for educational technology (including the development of two Domain Reference Models) is the primary contribution to new knowledge. This research is significance because it has potential to enable the synthesis of evaluation findings within the sector. This will enable an evidence-base of what works, for whom, in which contexts and why, ultimately benefiting policy-makers and practitioners to support better informed decision making and investment in education