Detecting DIF in polytomous item responses.

Use of polytomously scored items in educational tests is becoming increasingly common with new types of assessments such as performance assessment It is believed that performance assessment provides more equitable approaches to testing than traditional multiple-choice tests. However, some forms of performance assessment may be more likely than conventional tests to be influenced by construct-irrelevant factors, resulting in differential item functioning (DIF). Several methods have been proposed for DIF assessment with polytomous item responses, such as the Mantel procedure (Mantel, 1963), the generalized Mantel-Haenszel procedure (GMH) (Mantel & Haenszel, 1959), the polytomous extension of the standardization approach (STND) (Potenza & Dorans, 1995), the polytomous extension of SIBTEST (SIBTEST) (Chang, Mazzeo, & Roussos, 1996), and logistic discriminant function analysis (LDFA) (Miller & Spray, 1993). The purpose of this study was to investigate and compare the performance of the Mantel, GMH, STND, SIBTEST, and LDFA (LDFA-Uniform and LDFA-Nonuniform) procedures in detecting DIF with polytomous item responses under a variety of conditions. A simulation study was conducted to evaluate their Type I error and power in detecting DIF when the properties of the items were known. The factors considered were group ability distribution differences (three levels), test length (two levels), sample size (three levels), sample size ratio (two levels), item discrimination (null DIF, constant uniform DIF, and balanced uniform DIF) or item-discrimination parameter difference between reference and focal groups (nonuniform DIF) (three levels), and DIF conditions (four levels). Based on the findings of this study, all procedures had good and comparable Type I error control when groups were equal in ability or when groups had a small ability difference and item discrimination was not high or when groups had a large ability difference and item discrimination was low. Both LDFA-Nonuniform and GMH can be used to detect nonuniform DIF with LDFA-Nonuniform being more powerful when two groups had equal ability distributions and GMH being more powerful when two groups had different ability distributions. Finally, results from this study indicated that the LDFA procedure is preferable over the other procedures. (Abstract shortened by UMI.