The influence of evaluation on human service practitioner learning : an analysis of two case studies of practitioners' experiences of evaluation

Privatisation and the contracting out of human services to Non-Government Organisations (NGOs) is a feature of most countries with a strong ideological connection to neo-liberalism. The outsourcing of these services is accompanied with systems of performance management, along with program evaluation. Contemporary program evaluation emphasises the importance of collaborative and participatory processes to engage stakeholders, which enables program improvement, yet typically these evaluations seem to address the narrow accountability needs of service funders (Ebrahim, 2005a). Given that NGOs by and large have significant commitments to evidence based practice (Carman & Fredricks, 2008), and the improvement of program practice is in the interests of all parties, what are the conditions and barriers for practitioner learning from evaluation? This research involves two case studies of evaluations that aimed to foster practice learning, these will be explored in order to develop an understanding of the conditions and barriers to practitioner learning from evaluation. This research is novel in that it focuses specifically on practice level learning, an individually oriented phenomena that has received limited focus in the evaluation literature. The thesis draws on evaluation influence (Mark & Henry, 2004) as a framework to explore the broad and diffuse impact that evaluations can have, an approach well suited to exploring learning. Evaluation influence presents a series of mechanisms at the individual, interpersonal, and collective levels that reflect the full impact of evaluation and a cohesive way to organise theoretical and empirical knowledge of evaluations. Drawing on an understanding of the case, and the identification of influence mechanisms, the research sought to develop an understanding of the conditions and barriers to practitioner learning from evaluation. Analysis of these case studies suggests that there seems to be five key issues related to practitioner learning from evaluation. There was a lack of clarity in the relationship between evaluation and practice, despite there being organisational commitments to evidence based practice. There was a need to produce evaluation that had the validity to inform practice, evaluation with practice credibility as well as the kinds of validity and reliability understood by evaluators. Influence comes from the equitable and participatory engagement of practitioners in the evaluation, valuing their input and treating practitioners as stakeholders that possess important program and implementation knowledge. Acknowledging the critical politics of evidence was important in representing the practice context, and assuring stakeholders that the evaluation was not merely symbolic. Finally, managing the clash of inter-organisational priorities was imperative to getting practice learning on the agenda of an evaluation. Practice oriented evaluation (Schwandt, 2005) has the potential for significant benefits that translate to improvements in service delivery to isolated and vulnerable communities. This thesis adds to the concept of practice oriented evaluation by suggesting a number of important elements that it should include: embedding evaluation into practice; creating evidence that is valid to inform practice; making effective use of practitioners as evaluation stakeholders; engaging with the critical politics of organisations and the sector; and advocating for practice oriented evaluation at the planning stages of an evaluation