Problematizing the concept of the ‘borderline’ group in performance assessments

Many standard setting procedures focus on the performance of the ‘borderline’ group, defined through expert judgements by assessors. In performance assessments such as Objective Structured Clinical Examinations (OSCEs), these judgements usually apply at the station level. Our work analyses the assessment profile of OSCE candidates at the end of a 5 year undergraduate medical degree programme to investigate the extent to which the borderline group is consistent across stations. We look specifically at those candidates who are borderline in individual stations, and in the overall assessment. Whilst the borderline group can be clearly defined at the individual station level, our key finding is that the membership of this group varies considerably from station to station. These results raise real concerns with some commonly used standard setting methods, in particular, the borderline groups and objective borderline methods. They also suggest that institutions should ensure appropriate conjunctive rules to limit compensation in performance between stations. Finally, we argue that our results provide further arguments in favour of a sequential testing format for OSCEs. In comparison with a one-size-fits-all model for OSCEs, this allows assessment of ‘borderline’ candidates across a wider range of content areas with concomitant improvements in pass/fail decision-making