Using Generalizability and Rasch Measurement Theory to Ensure Rigorous Measurement in an International Development Education Evaluation

Between the United States and Great Britain, over 30 billion USD was spent in 2018 on international aid, over a billion of which is dedicated to education programs alone. Recently, there has been increased attention on the rigorous evaluation of aid-funded programs, moving beyond counting outputs to the measurement of educational impact. The current study uses two methodological approaches (Generalizability (Brennan, 1992, 2001) and Rasch Measurement Theory (Andrich, 1978; Rasch, 1980; Wright & Masters, 1982) to analyze data from math and literacy assessments, and self-report surveys used in an international evaluation of an educational initiative in the Democratic Republic of the Congo. These approaches allow the researcher to identify and select pertinent facets and look at them in relation to one another, allowing us to attribute smaller or larger sources of variability to a particular facet, and using both provides additional insight to instrument development and validation efforts. A thorough analysis of five Early Grades Reading Assessment subtasks, five Early Grades Mathematics Assessment subtasks, and three sets of items from a survey administered to the girls in the study was completed. Results suggest that two factors were consistently flagged as contributing to error in the outcome measures: enumerators and language of administration/girl’s home language. The results of this study provide implications for several phases of evaluations of educational initiatives in developing countries: evaluation design development; the importance of a pilot in assisting in refining the design and sampling plan; and the importance of selecting the appropriate outcome measure, particularly in projects utilizing payment for success models. The results also indicate the utility and complementary nature of using Generalizability and Rasch Measurement Theory analytic procedures in assessing the quality of complex evaluation data. Evaluations such as the one used in this study are highly complex in nature, with more possible sources of error than those included in the current study. What these results indicate is that though there is a wish to standardize and assess in difficult settings, the fact that context affects not only the results of assessments like the EGMA and EGRA, but their utility, cannot be ignored