Translating theory to practice: a multi-method study of the contribution analysis evaluation approach

As the evaluation discipline moves away from the “black-box” evaluations, theory-based evaluation approaches such as Contribution Analysis (CA) have gained popularity. This study responded to explicit requests to probe deeper into CA (e.g., Budhwani & McDavid, 2017; Delahais & Toulemonde, 2012; Dybdal et al., 2012; Lemire, 2010) as well as a more general call for a systematic examination of evaluation cases in order to generate “practical knowledge (Schwandt, 2008) about the Contribution Analysis (CA) evaluation approach (Mayne, 2012). As such, this dissertation employed a multiple-method design of two segments, Phase I consisted a systematic review of the conceptual literature, and the ensuing Phase II called upon a multiple case study of seven empirical CA cases. Taken together, the study design allowed for a systematic inquiry into the theoretical translation and practice of contribution analysis (CA). Specifically, the study investigated how Contribution Analysis is conceptualized by theorists, and how this is understood and translated into practice by evaluation practitioners. In this endeavor, the dissertation identified elements of effective practice by characterizing adaptations, adjustments, and innovations, and identified conditions under which practices may be different (e.g., evaluand, contexts, or evaluators) (Smith, 1993). The study pursued a richer understanding of the contexts of practice and to probe deeper into the translation of theory to practice using empirical evidence to further develop and improve on the current understandings of CA theory and practice. The findings from this study contribute to the empirical body on evaluation theory and practice, as well as methodological contribution on conducting research on evaluation. Moreover, findings from this study seek to inform the development of CA contingency theories, which identify conditions under which practices are effective, which are considered to be the strongest types of evaluation approaches as they are buttressed by empirical knowledge of practice (Shadish et al., 1991; Smith, 1993). The dissertation findings illustrate the complexity of the contexts in which CA evaluations take place, and identified contextual factors related to the program theory, sector of practice, geographic scope, temporal interval, and the effect-object. It seems that for certain contexts and purposes, CA by itself may not be enough as it not amenable for the comparison of causal packages across contexts nor cases. Therefore, approaches like Process Tracing (PT) or Bayesian modeling, or Qualitative Comparative Analysis (QCA) are used to facilitate direct comparisons of effects/influences of impact pathways from case to case. I argue however, that the understated value of Contribution Analysis is in its function as a validation approach, which creates a framework to build robust and sound arguments in support of contribution claims