Evaluation as a multi-ontological endeavour: a case from the English National Program for IT in healthcare

This paper analyzes how researchers’ different ontological and epistemological assumptions shape the process and outcomes of evaluation research. Focusing on the critical realism (CR) and social constructionism (SC) philosophical approaches, it outlines the rationale for multi-ontological evaluation and develops principles for conducting it. The paper draws from experience of evaluating a national implementation program of electronic health records in hospitals, one of the projects of the English National Programme for IT. It argues that an evaluation based on SC and one based on CR are significantly different in how they use knowledge gained in the field, and in the kind of evidence and recommendations that they can offer policy makers. The CR philosophy applied to evaluation research provides foundations from which judgments and abstractions can be presented in the form expected by the policy makers and managers who commission evaluations. In line with its ontological standing, social constructionism cannot simply or directly abstract and generalize across contexts, though it can offer other types of valuable evaluative insight. We show that, despite their differences, these two philosophical positions can, when taken together, produce jointly useful knowledge. This paper argues for the use of multi-ontological evaluation approaches and provides guidelines for undertaking such endeavors by emphasizing the need for mutual respect, dialogue, negotiation, and reflection