Comparison of three scoring procedures within context-dependent item sets

It has been theorized and evidenced that traditional reliabilities calculated on tests consisting of context-dependent item sets yield inflated estimates. However, the degree of inflated reliability for different scoring techniques has not been observed. This study scored three context-dependent item sets as stand-alone items and as separate item sets using three different scoring techniques; number-right, polyweighting and the three-parameter IRT logistic model. Differences in reliability estimates for the stand-alone and item set treatment were determined for each scoring procedure and compared. These three scoring procedures were also compared to determine which procedure yielded the highest reliability and validity estimates and precision of scoring within the item set treatment. The findings of this study were inconsistent with previous research in that only the polytomous scoring technique yielded inflated reliability estimates. In every comparison, number-right scoring and polyweighting were similar and outperformed the three-parameter IRT ability estimation model