Improving the validity and reliability of OSCE scores through a comprehensive measurement instrument

Background: As an integral tool for clinical skill assessment, the quest to enhance the validity and reliability of OSCE scores should be ongoing. Summary of work: A new instrument with a wide spectrum of scales that allow objectivity and precision (i.e. checklist, process performance rating, and an overall global rating) was introduced, replacing the use of holistic numerical scoring. The new marking format was piloted and used in nine OSCE stations for first year students and 14 OSCE stations for second year students in 2010. It was also specifically designed for computerized scoring using Optical Mark Recognition software. Summary of results: Findings suggest that the new instrument is well accepted by examiners for its ease of use and enhanced objectivity. Empirical evidence also indicates better discrimination for students’ clinical skills, stronger internal consistency, greater inter- and intra-station reliability, and improved individual examiner marking consistency. Conclusions: The validity and reliability of OSCEs can be enhanced with the use of practical scoring scales that offer examiners greater objectivity and precision. Take-home messages: The procedural nature of the OSCE should not preclude the incorporation of scales targeting various levels of objectivity and precision to enhance the validity and reliability of student results