An exploratory study of cognitive laboratories for development and construct validation of reading and mathematics achievement test items

This study explores the usefulness of the cognitive laboratory, and specifically of the think aloud procedure, for development and construct validation of achievement test items in fourth grade reading and eighth grade mathematics. While cognitive laboratory methods have been used over the past two decades to reduce the measurement error associated with surveys, this study represents the first systematic use of cognitive laboratories in developing achievement test items. In the research presented here, 430 students participated at six sites nationwide in the cognitive laboratories. The self-selected sample of students included 234 fourth graders and 196 eighth graders. Five hundred and eighty-four test items were evaluated in the laboratories, including multiple choice, short constructed response, extended constructed response, and gridded response items. Analyses of data from these laboratories found that the cognitive labs were able to identify a variety of situations in which items were not measuring the cognitive processes or skills the items were intended to measure, with significant differences found based on item format. Multiple choice items were less problematic than extended constructed response items. The most common threat to construct validity identified in the cognitive laboratories for math and reading items was language difficulty. The information provided by the cognitive labs supplements information gathered in the subject matter expert reviews. The quality of interviewers who led the cognitive lab sessions ranged widely, but interviewer quality did not appear to have a discernable relationship with the quality of student think alouds, or with student performance on test items. It is hoped that the results from this study will lead to the further refinement of cognitive labs as a technology for improving the construct validity of tests in reading and mathematics