Unidimensional Vertical Scaling of Mixed Format Tests in the Presence of Item Format Effect

The purpose of this study was to contribute to the existing body of evidence on vertically scaling mixed format tests by examining the impact of item format effect in conjunction with specific configurations of common item sets on two of the most popular calibration methods under test specification and scaling scenarios likely to exist in practice. In addition to advice for practical application provided by the investigation, this study also explored the impact of explicitly modeling the vertical scale factor when simulating data compared to a traditional model for in which the underlying vertical scale is implied.\ud Using a CINEG data collection design, six grade level tests, consisting of 61 items, were created with a 9:1 ratio of multiple-choice to constructed-response items and two different sets of 14 mixed format items designated as common items. Ability distributions for 2000 students per grade level were generated with the mean ability for successive grade levels increasing at varying increments to simulate grade level separation along with four covariance structures that reflected varying degrees of correlation to simulate item format effects. \ud Under a 3PL-GRM model combination, expected scores were calculated with recovery of expected score used as evaluation criteria. Ability and item parameters were estimated using the MLE proficiency estimator in MULTILOG and transformation constants were calculated in STUIRT using the Stocking-Lord linking method. The performance of separate and pairwise concurrent calibration was then examined by calculating average BIAS and average RMSE values across 100 replications.\ud While the results of the study provided evidence that item format effects, vertical scaling method, and separation between grade levels significantly impacted the vertical scales, influence of these variables was often in combination with one another. The major findings were (1) pairwise concurrent calibration holistically performed better compared to separate calibration; (2) moderate to large item format effects were more likely to bias resultant vertical scales; (3) a large separation between grade levels resulted in a more biased vertical scale; and (4) explicitly modeling the vertical scaling factor during data generation influenced mean RMSE values more significantly than mean BIAS values. \u