A simulation study to investigate optimal equating anchor set construction practices under the NEAT design

This study examines anchor set construction techniques in observed score test equating under the non-equivalent with anchor-test design. It differs from other studies in that it seeks to understand the interaction between the examinee abilities, test specifications, and anchor set properties and develop a set of construction guidelines for practitioners. This simulation study includes achievement and certification testing scenarios, each with 48 total difficulty and discrimination alignment conditions for the overall test and anchor set. Six candidate ability distributional conditions represent situations where the alternative test form candidates are either more able, more homogeneous, or more able and more homogenous compared to the base form group. In this study all 576 test, anchor, and ability conditions are equated by two linear methods, Tucker and Levine Observed Score, and two nonlinear methods, Frequency Estimation and Equipercentile. The results of this study identified three interactions that are important to consider when equating and are not impacted by anchor set design: 1) well aligned test forms and similar examinee groups, 2) similar examinee groups and off-target test forms, and 3) well aligned test forms and dissimilar examinee groups. The results also suggest that three conditions are important and are impacted by anchor set design: 1) off-target test forms and dissimilar examinee groups, 2) well aligned test forms and more homogenous examinee groups, and 3) off-target test forms and more homogeneous and dissimilar examinee groups. The second objective of the study is to develop a set of construction techniques for practitioners to use when dissimilar examinee ability groups are expected. The results suggest that the Equipercentile and Levine equating methods produce the smallest amount of systematic and total equating error when examinee groups differ in ability, are more homogeneous, or differ in ability and are more homogeneous. Two specific anchor set construction techniques are recommended for use with the Equipercentile method: 1) a midi anchor set with increased discrimination or 2) a midi anchor set with increased difficulty and discrimination. The results suggest that the Levine method is the most flexible, particularly if a more homogeneous sample is expected. Specifically, two types of anchor sets are recommended for use with the Levine method when a more able and more homogeneous sample is expected: 1) an anchor set with increased difficulty for low discrimination tests and 2) a traditional mini anchor set for high discrimination tests