Analysis of MCQ and distractor use in a large first year Health Faculty Foundation Program: assessing the effects of changing from five to four options

Background: Multiple choice questions are commonly used in summative assessment. It is still common practice for tertiary institutions and accrediting bodies to use five-option single best answer multiple choice questions, despite a substantial body of evidence showing that multiple choice questions with only three or four options provide effective and discriminatory assessment. Methods: In this study we investigated the distribution of distractor efficacy in exams from four large first-year undergraduate courses in chemistry and in anatomy and physiology in a Health Faculty; assessed the impact on overall student score after changing from five-option to four-option single best answer multiple choice questions; and assessed the impact of changing from five options to four options on item difficulty and discrimination. Results: For the five-option questions analysed, 19% had four effective distractors, which is higher than previous studies, but still a minority of questions. After changing from five to four options, the overall student performance on all multiple choice questions was slightly lower in the second offering of one course, slightly higher in the second offering of another course, and similar in the second offering for two courses. For a subset of questions that were used in both offerings, there were negligible differences in item difficulty and item discrimination between offerings. Conclusions: These results provide further evidence that five-option questions are not superior to four-option questions, with reduction to four options making little if any difference to overall performance, particularly when MCQ is used in conjunction with other assessment types (including short answer questions, and practical or laboratory assessment). Further areas of study that arise from these findings are: to investigate the reasons for resistance to changing established assessment practice within institutions and by accrediting bodies; and to analyse student perceptions of the impact of a reduced number of options in MCQ-based assessment.Full Tex