Making principled decisions about curriculum development: outcomes of a Realist evaluation across 13 universities

Curriculum change is often driven by external factors such as changes in the field or informal feedback from students and staff rather than systematic data collection [1, 2]. Over the last five years nearly all of the universities in Australia and New Zealand have introduced projects based on the Engineers Without Borders (EWB) Challenge. Every year, EWB nominates one of their partner organisations in a developing community and a range of projects and themes addressing needs and work in that community as the basis for the year's EWB Design Challenge. EWB develops and provides a suite of resources including on-line information about the community and the partner organisation's work which is provided for use in first year engineering courses. In 2010 the Australian Learning and Teaching Council funded us to evaluate this innovation across 13 universities in Australia and New Zealand. All of the partner institutions have implemented this innovation differently and comparison of these different implementations affords us the opportunity to assemble "a body of carefully gathered data that provides evidence of which approaches work for which students in which learning environments" [2]. Data was collected through observation of classes, interviews and focus groups with staff and students, analysis of documents such as course outlines and student work and an exit survey offered to all participating students (N = approx. 4500). This large body of data has been analysed using a Realist approach [3] to isolate the aspects of Context and the Mechanisms that are triggered by these environments in order to produce the observed outcomes. We argue that curriculum design can be improved by close attention to factors like these, which are empirically derived, rather than on impressions and assumptions or the latest educational fad