Exploring the intersection of ethics and method in evaluation practice : an MDS study of compatibility and conflict within The Program Evaluation Standards (2nd edition)

Researchers and evaluators frequently perceive pressure from numerous stakeholders, their own philosophical positions, and methodological dictates requiring trade-offs in method, scope, and professional practice. The Program Evaluation Standards have, therefore, been developed to enhance the quality and fairness of evaluations by providing a guide for evaluating educational programs. However, the Standards are not prescriptive in nature and require substantial professional judgment and skill to apply appropriately. Compromise, even among which Standards to implement and which Standards to sacrifice, remains a necessary part of evaluation practice. Using the Standards as a framework for inquiry, this study seeks to . understand the underlying cognitive dimensions along which professional evaluators make tradeoffs in practice. Evaluators were surveyed and asked to rate the compatibility and conflict between all pairs of Standards, and a multi-dimensional scaling analysis was conducted. The results confirmed earlier findings that political considerations within the system of evaluation stakeholders broadly defined tend to have the greatest impact on the trade-offs made. The old divisions of qualitative versus quantitative inquiry are largely breaking down and blended approaches are being used to create compromises while minimizing the sacrifices to quality, fairness, and ethical practice where possible. The emerging dimensions included: 1) robust collection and reporting of qualitative information vs. what is practical within the constraints presented, 2) responsibility to decision makers vs. responsibility to broader stakeholder groups, 3) research ideals vs. what is sufficiently good practice, and 4) humanist research principles vs. traditional scientific principles.Education, Faculty ofEducational and Counselling Psychology, and Special Education (ECPS), Department ofGraduat