Using multiple response true-false multiple choice questions

The Royal Australasian College of Surgeons (RACS) Board of Examiners, having agreed to replace Type K with Type X multiple choice questions (MCQ), undertook a study to determine the most appropriate method of scoring the latter. Retrospective scoring of candidates’ responses in a previous examination, using six reported scoring techniques for Type X items, showed that the Middlesex technique yielded a mean score closest to the mean score of the Type K items which were being replaced. However, as candidates’ responses are affected by instructions, and the latter could not be controlled, a prospective study was carried out on three examinations, in each of which the nature of Part C varied among the three papers, while Parts A and B were similar. Paper 1 contained Type K items in Part C, Paper 2 contained Type X items with instructions appropriate for the Middlesex technique, and Paper 3 contained Type X items scored +1 if correct, –1 if incorrect and 0 if omitted. Items sampling the same content areas were randomly distributed among the three papers. The results indicated that the two Type X systems yielded significantly lower mean scores than Type K, probably due to combinations characteristic of the latter, enabling candidates with partial knowledge to obtain a maximum score. The difference in mean scores between the two Type X systems was not significant. A significantly higher mean percentage score in reused Type X (Middlesex system) items over that of the same items when they were previously used as Type K items was attributed to the combined effects of credit for partial knowledge, practice on reused items and differences in mean candidate ability across years. Pearson product moment correlation coefficients between pairs of Part C mean percentage scores at each examination were positive and significant. The Board decided to replace Type K with Type X items, and the +1, 0, –1 system of scoring was adopted as it was considered more efficient than the Middlesex system, which did not give credit for identifying false statements as false