Enhancing Ability Estimation with Test-taking Strategies

A central purpose of the field of educational assessment is to estimate the ability of test takers. Many large-scale national and international assessments are low-stakes assessments, but some examinees do not take these tests seriously, displaying suboptimal testing behavior such as rapidly guessing and skipping (i.e., omitting) items. This “bad” test-taking behavior can bias parameter estimates, unfairly disadvantage examinees who carefully complete all items (Braun, Kirsch & Yamamoto, 2011; O’Neil, Sugrue & Baker, 1995), and reduce the consequential validity of the ability estimates (McKnight, McKnight, Sidani & Figueredo, 2007). ^ Most large-scale assessment programs measure ability with correct/incorrect scores, but theory and the broader literature on measuring ability indicate that correct/incorrect information is only one facet of ability. In fact, compared with traditional Item Response Theory (IRT) models that are typically used to estimate ability, it has been shown that some IRT models that have been developed to incorporate either response time or omitting reduce bias and improve estimation of ability (Klein Entink, Fox, & van der Linden et al., 2009; Moustaki & O’Muircheartaigh, 2000; O’Muircheartaigh & Moustaki, 1999; Wang & Hanson, 2005). But, large-scale assessment programs have yet to use these models, instead preferring to use traditional IRT measurement model plus covariates to reduce bias (Mislevy, Johnson, & Muraki, 1992). Thus, although patterns of scores, response times, and response indicators provide unique information about examinees’ ability levels, the approach is unsatisfactory. Yet, literature on response times is increasing in recent years, indicating a renewed willingness to consider response time methods and models (Kyllonen & Xu, 2016). ^ The purpose of the present study is to investigate another method to incorporate test-taking strategies into estimation to reduce bias in the ability estimates: using a summary of test-taking behavior as covariates. Two studies were conducted to explore variations on the approach. In Study 1, the approaches are applied to a real data set, and in Study 2, the approaches are evaluated using Monte Carlo simulation methods.