An empirical investigation of the robustness of the Mantel-Haenszel procedure and sources of differential item functioning

This study was intended to empirically evaluate the reasonableness of the Mantel-Haenszel procedure used to appraise differential item functioning or item bias and to investigate item characteristics which results in differential performance for the gender and ethnicity comparison.This study, with real data, examined four major research questions: (a) How stable are the Mantel-Haenszel statistics across different samples? (b) Does the Mantel-Haenszel procedure confound mean differences between focal and reference groups with differential item functioning? (c) Can an additional matching variable reduce the number of biased items? (d) Are ethnicity and gender differences on a math item related to item characteristics such as cognitive skill and content category of item?The data for this study were response data from the Illinois Mathematics Assessment Test used in 1991 Illinois Goal Assessment Program (IGAP). Samples for analyses are confined to the eleventh grade.Major findings of this study are: (a) The Mantel-Haenszel statistic is quite stable across different samples when the standard error of the statistic is taken into account. (b) The Mantel-Haenszel procedure is quite robust against discrepant ability distributions. In other words, the M-H procedure does not confound group mean differences with DIF. (c) The Mantel-Haenszel multiple conditioning design can be used to minimize the confounding effect of an additional conditioning variable, but the multiple conditioning design does not necessarily reduce the number of DIF items. (d) Item characteristics examined in this study are significant predictors of differential performance on the gender and ethnicity comparison. Gender and ethnicity differences were observed in several item categories.It may be said that the result from this study provides a strong support for the reasonableness of the Mantel-Haenszel procedure. The identification of item characteristics causing differential performance on math items may contribute a better understanding of gender and ethnicity differences in learning mathematics.U of I OnlyETDs are only available to UIUC Users without author permissio