A critical examination of the functions and methods of evaluation research

This thesis examines the aims and methods of evaluation research, and, ultimately, considers what functions evaluation may serve in innovative programs. It addresses the controversy with respect to the inadequacy of the scientific experimental method in evaluating social action. Analysis of a case study indicates that applying the experimental method is problematic when programs have vague concepts and uncertain targets, which is almost certain in innovatory programs. Even if that method were successful in generating reliable facts about program effects, it would have overlooked some important subjective aspects of the program. Our general conclusion suggests that it is important to go beyond scientific experimental approaches by exploring alternative modes of evaluation. Critical theory, it is suggested, is useful for studying the normative and latent aspects of programs and program effects. A main presupposition of this study is that an epistemological investigation, which examines the theories of knowledge underlying the aims and methods of evaluation, can make a worthwhile contribution to program evaluations. Inherent in evaluation are fundamental epistemological questions such as "What may be considered knowledge?", "What is the factual and value content of that knowledge?", and "For what purposes do we use enquiries?". Numerous figures in evaluation suggest that not enough attention is given to the conceptual analysis of evaluation purposes and methods. The literature on evaluation is the source of our analysis of the functions and methods proposed by various practitioners. We examine the views of two prominent camps: those who support the scientific experimental methods of testing, measuring and generating facts; and those who use normative social theories and attend to value issues and subjective aspects of programs. Relying upon the literature related to social enquiry, the thesis examines diverse theories of knowledge: scientism, positivism, pragmatism, and finally, critical theory. The critical school advocates a flexible approach to enquiry which extends beyond scientific experimental methods. Following Kant, it emphasizes the pervasive influence of a priori cognitive models in our generation of knowledge. It recognizes, too, the importance of the Hegelian view that the history of abstract thoughts and values is essential for understanding social affairs. As well, its critique incorporates the Marxian stance of focusing on the conflicts of interest among social classes. Accordingly, it goes beyond the positivistic notion that bare facts devoid of value content can be generated. The critical approach, by considering multiple perspectives and by investigating latent aspects such as value systems, conflicting interests, perceived social needs and human consciousness, is appropriate for studying the complex nature of social action. Following upon these concerns, the following evaluation functions are proposed: to clarify theoretical assumptions of programs, to reveal and revise value assumptions and relate them to a history of ideas, to identify the social impacts of programs and the perceived needs of program participants, and ultimately, to foster the exercise of critical self-reflection among participants by encouraging decentralized decision-making and evaluation. These criteria are applied to a case study, an evaluation of a community energy program. The conclusions drawn from the case study indicate that the controlled experiment should be supplemented with subjective evaluation approaches. The experiment may be useful at times, but at least with respect to social innovative projects, its application is problematic. With the presence of program features including vague program objectives, uncertain criteria of program success, and unvalidated causal assumptions (attitude →behaviour → actual energy savings) , it is impossible to "test" (in the scientific sense) hypotheses such as Program A →Causal Process B → Objective C. Our critical examination suggests that, by concentrating on concrete program effects, the case study evaluation had overlooked important intrinsic factors affecting the program and plan. These factors are the program's theoretical and value assumptions, the perceived needs of recipients, and the conflicts of interest between the sponsors and the program designer. The critical school of thought reminds us that we have to persistently question the factual content, value basis and causal assumptions of our programs. Critical theory, by identifying the limits of traditional enquiries and exploring alternative epistemologies, provides invaluable insights for developing guidelines for evaluation research.Applied Science, Faculty ofCommunity and Regional Planning (SCARP), School ofGraduat