Using on-line examiner training to improve inter-rater reliability

Background: A crucial determinant of reliability in OSCE is the accurate judgment made by the examiners, particularly if the pass/fail decision is made by a single examiner. Attempts to improve the reliability of OSCEs include organising training sessions to allow examiners carry out their role consistently thereby reducing variability in scoring. However, due to the busy schedule of clinicians and the challenges of getting away from their activities to attend examiner-training sessions, the validity and reliability of the examinations could be compromised. Description: An OSCE collaboration project was developed between two Australian Universities in 2010 in which three OSCE stations were developed and embedded in the first clinical phase examinations at both schools. To reduce variability in scoring, an OSCE e-scoring tool was developed and set up in a secure on-line Blackboard Learning System Vista environment. The three shared OSCE scenarios were videotaped and used for on-line examiner training. All internal and external examiners were invited one week prior to the examination via email, given login access and instructions on how to use the program. In their own time, each examiner was able to watch and assess two unlabelled scenarios (poor and good performance) of the OSCE case which they had been assigned to examine on. After completing and submitting their scoring sheets, the examiners were able to compare scores, reflect on their judgments and discuss their decisions on-line. They were also asked to provide feedback on their experiences of the e-scoring program, using an on-line survey. Results: The e-scoring package gave the examiners the opportunity to standardise their marking by comparing their scores with their co-examiners and reaching consensus on scoring techniques. Similar trends in the results were observed at both schools with high inter-rater reliability, especially with the global scores. Examiners valued the process as it allowed them to set the 'expected standard' for the station prior to the actual exam. They also indicated that this sort of tool should be used more widely in OSCEs. Applicability: The observed close agreement between examiner scores in this study, despite the different geographical locations is attributed to the e-scoring program because it offered training exercise for both quality assurance and appraisal purposes. The efficacy and ease of use of this novel approach to examiner training indicate the possibility of its wider use in OSCEs. Adaptability: The importance of the commitment of medical educators to the quality assurance of OSCEs cannot be overemphasised. Results from this study revealed that the e-scoring program has the potential to enhance inter-rater reliability in OSCEs. With increasing student numbers, numerous teaching sites within each medical school as well as time constraints, this tool will afford time-poor clinicians the opportunity to better engage with the assessment process and reach consensus on their scoring techniques, thereby providing validity evidence to all stakeholders