Assessing class-wide consistency and randomness in responses to true or false questions administered online

We have developed simple data-mining algorithms to assess the consistency and the randomness of student responses to problems consisting of multiple true or false statements. In this paper we describe the algorithms and use them to analyze data from introductory physics courses. We investigate statements that emerge as outliers because the class has a preference for the incorrect answer and also those that emerge as outliers because the students are randomly changing their responses. These outliers are found to include several statements that are known in the literature to expose student misconceptions. Combining this fact with comments made by students and results of complementary assessments provides evidence that the tendency of a group of students to change their answer to a true or false statement or to remain consistent can serve as indicators of whether the class has understood the relevant concept. Our algorithms enable teachers to employ problems of the type described as a tool to identify specific aspects of a course that require improvement. They also enable researchers to employ such problems in experiments designed to probe aspects of students’ thought processes and behavior. Additionally, our results demonstrate that at least one category of research-inspired problems (ranking tasks) can be adapted to the linked true or false format and productively used as an assessment tool in an online setting