1Running head: COGNITIVE LABS AND TEST DEVELOPMENT The Applicability of the Cognitive Laboratory Method to the Development of Achievement Test Items

This study explores the utility of the cognitive laboratory, and specifically the think aloud technique, for the development of achievement test items in 4th grade reading and 8th grade mathematics. While cognitive laboratory methods have been used over the past two decades to reduce the measurement error associated with surveys, this study represents the first systematic use of cognitive laboratories for test development. Four hundred and thirty students participated in the cognitive laboratories, which were held at six urban and suburban sites across the country. The self-selected sample of students included 234 4th graders and 196 8th graders. Five hundred and eighty-four test items were evaluated in the laboratories, including multiple choice, short constructed response, extended constructed response, and drawn or gridded response items. We found that a large proportion of the items were problematic; 80 % of reading items and 74 % of mathematics items exhibited problems. The most common problem types identified in the labs were problems with unclear language or contexts, and unclear or incomplete scoring rubrics. The cognitive labs, when used in combination with expert reviews, appear to provide significantly greater information on item problems than either method alone in mathematics. In both math and reading, the combination of cognitive labs and expert reviews resulted in greater identification of typographical and text editing problems. The cognitive labs were also especially useful at identifying language/contextual and rubric problems in mathematics items. These findings show that the cognitive laboratory method and the think aloud procedure appear to be useful tools for studying the processes and skills used by students when answering test questions