Higher Order Testlet Response Models for Hierarchical Latent Traits and Testlet-Based Items

Both testlet design and hierarchical latent traits are fairly common in educational and psychological measurements. This study aimed to develop a new class of higher order testlet response models that consider both local item dependence within test-lets and a hierarchy of latent traits. Due to high dimensionality, the authors adopted the Bayesian approach implemented in the WinBUGS freeware for parameter esti-mation. A series of simulations were conducted to evaluate parameter recovery, consequences of model misspecification, and effectiveness of model–data fit statistics. Results show that the parameters of the new models can be recovered well. Ignoring the testlet effect led to a biased estimation of item parameters, underestimation of factor loadings, and overestimation of test reliability for the first-order latent traits. The Bayesian deviance information criterion and the posterior predictive model checking were helpful for model comparison and model–data fit assessment. Two empirical examples of ability tests and nonability tests are given. Keywords item response theory, testlet response theory, higher order models, hierarchical latent trait, Bayesian methods Testlet design, in which a set of items share a common stimulus (e.g., a reading comprehension passage or a figure), has been widely used in educational an