How monitoring and evaluation systems support (or fail to support) organisations to improve their performance

Evaluation can be a useful tool for organisations to examine whether programs and policies work, and how they work for different people in different circumstances. The interest in evaluation is widespread, across sectors and types of organisations, and there is much evaluative activity and requirements for evaluation in international development. However, while evaluation can be useful, its effectiveness can be limited. Evaluations can be commissioned for political purposes, evaluation commissioners may not agree with findings or evaluation methods, resources to implement findings may be inadequate or findings may come too late – after the program or policy has ceased. Additionally, evaluations are generally conducted on single programs, while organisations now need to understand their effectiveness across multiple interventions or thematic areas. As a consequence, there has been a growing interest in building monitoring, evaluation and learning (MEL) systems to address these limitations (Liverani & Lundgren, 2007; Simister, 2016). The idea is that MEL systems draw from monitoring data and individual evaluations to reach conclusions about program and organisational effectiveness. However, there is a lack of clarity about how these MEL systems work or how they might work. This thesis analyses the learning systems, a key part of the MEL system, of two Melbourne-based international non-government organisations (INGOs). The organisations set up and sanctioned staff to use the two learning systems in order to better understand and improve their practice, and for each organisation to encode relevant experience into improved policies and procedures. Both organisations developed these systems in the context of an increased emphasis on the impact of aid and development assistance by global development institutions and Organisation for Economic Cooperation and Development (OECD) donors. Developing learning systems was one way the two INGOs increasingly focused on assessing and enhancing their development effectiveness in order to meet their own and their international donor expectations. The thesis uses the Kirkpatrick model of training evaluation, alongside concepts of evaluation use, to assess these two learning systems, including their conception, structure, process and outcomes. The Kirkpatrick model investigates the effectiveness of training by examining outcomes across four levels. The first level is whether trainees find the training high quality and relevant, the second level is whether trainees learn, the third level is whether trainees implement their learning and the fourth level is whether the implementation of learning leads to an improvement in organisational results. The study finds that both systems were effective at levels 1 and 2. Staff valued the learning activities and reported learning from their involvement. However, the systems were less effective at converting learning into behaviour. That is, most staff did not change their practice because of what they had learned in the systems. And, given the lack of wide-scale change in practice, the organisational learning systems were unable to impact on organisational results. The thesis argues that evaluation and learning needs to better support practice improvement and thereby increased organisational effectiveness to justify the resources it consumes. In order for MEL systems to support improvements in practice, they must separate learning-oriented evaluative and organisational learning activity from upwards-focused accountability exercises. Learning oriented activity must be carried out using approaches and strategies that foster behaviour change. The model developed in the thesis, EPoC – Embodied, Power-conscious and Collective – comprises three learning approaches and associated strategies for evaluators and organisational learning facilitators to use when developing MEL systems. The EPoC approach to learning supports people to work through the implications for how what they learned will change how they behave rather than only focusing on transmitting knowledge. EPoC directly addresses power dynamics that affect how data are produced, shared and used rather than being blind to power imbalances and inequalities, particularly gendered and racial inequalities. EPoC acknowledges that all behaviour change in organisations involves groups of people. For this reason, EPoC understands learning as improved collaborative practice rather than individual cognitive change. The EPoC approach was developed through case studies of two INGO two learning systems and is most relevant to organisations that are working towards positive social change. The model can also be tested by organisations in other sectors working to improve performance, through supporting practitioners change their behaviour, for relevance