Validity and valuing in program evaluation: case studies of practice

© 2000 Dr. Delwyn Anne Evelyn GoodrickProgram evaluation is a relatively new profession charged with a mission to assess the value of educational and social interventions. It is evolving rapidly and progressing through the growing pains of defining its central tenets. Confusing this growth are various epistemological and training orientations, with differing perspectives about valid claims, making common ground in theory and practice elusive. Traditional conceptions of validity may be useful in understanding the validity of factual claims, but they are inadequate for understanding and assessing evaluative claims. The main contribution of this thesis is its attempt to establish valuing as an incontrovertible aspect of evaluation, and to present a schema that evaluators from any epistemological or training orientation may use to enhance their consideration and communication of valid evaluation claims. This thesis contains a theoretical review of validity in social science research. Four important epistemological perspectives are presented to explicate this theoretical overview. These perspectives are reflected in formal models or approaches to program evaluation. Five evaluation theories are presented to highlight the various purposes of evaluation, and perspectives on validity and valuing. They are theory-grounded approaches, fourth generation evaluation, deliberative democratic evaluation, utilization-focused evaluation, and advocacy approaches. Three interpretive case studies formed the empirical component of this study. The cases represented short to mid-term evaluations conducted by external evaluators. The three cases involved an evaluation of an innovative model of health care in a hospital setting, an evaluation to assist school administrators to understand the reasons for declining enrolments, and an evaluation of the educational needs of rural students. The instrumental case studies illustrated the salience of valuing, the difficulties associated with generating evaluative claims and the lack of explicit criteria to determine the validity of value claims. This thesis extends validity theory with a presentation of a heuristic schema that evaluators from any epistemological orientation may use to enhance their consideration and communication of claims in evaluation. The schema is composed of three dimensions of evaluation that are presented for consideration to maximise the validity of claims. These dimensions are, a) the level of engagement in the project being evaluated, b) the degree of reliance on method-structure, and c) the degree of value inclusion. Each of the dimensions is presented as a continuum with the extreme positions at either pole representing threats to the validity of evaluation claims. The evaluators' positioning on the three dimensions in an evaluation requires reflection, argument, and ideally negotiation with clients to justify the stance on the three components. It is not possible, nor desirable, for evaluators to become homogeneous in theory and application. Evaluators need to become more aware of the issues that impinge on the validity, utility, and clarity of their work. It is hoped that this thesis provides a stepping stone toward this purpose