Feedback after summative OSCEs: optimising delivery and engagement

Background Few opportunities for learning exist after high stakes summative assessments. Providing students feedback after summative Objective Structured Clinical Examinations (OSCEs) could be a learning opportunity. However, as engagement is often poor, it is necessary to have an understanding of the learners’ perspective to design a summative feedback programme. Once summative feedback has been given, it is necessary to evaluate its quality and quantity and whether the learners engage with it as the examiners and learners may have little motivation and opportunity to engage with summative feedback. Evaluating whether the learners read the feedback they receive belongs to the behavioural aspect of feedback engagement. Looking beyond this, cognitive and emotional aspects of feedback engagement can be better understood by defining and measuring specific learner attitudes that influence feedback engagement. Currently, there is no evidence of such constructs in medical education. Feedback orientation is a construct that represents a learner’s receptivity to feedback. It is composed of Utility, Accountability, Feedback self-efficacy and Social Awareness dimensions. Feedback Orientation Scale is a twenty item five-point questionnaire that measures these constructs in business management. This study aims to generate the evidence of feedback orientation in medical education. Methods This is a mixed methods study conducted at the University of Nottingham. Firstly, a qualitative phenomenological study was conducted using semi-structured peer interviews to explore the learners’ perception of feedback in the context of summative assessments in order to inform the design of the intervention. Secondly, a feedback intervention was implemented in which examiners used digital tablets to record written feedback to students during actual high stakes summative OSCEs. This feedback was delivered online at the same time as the results. The quality of the comments was assessed using a set criteria. Feedback website log-in data were collected. Learners were categorised into Honours, Pass and Fail groups. Quantity distribution of feedback across these groups and their log-in rates were analysed. Thirdly, confirmatory factor analysis, item analysis and ANOVA were performed on FOS data collected from medical students. FOS scores were analysed in relation to the learners’ course, training year and gender. Results Interviews The students (N = 16) wanted specific personalised examiner feedback to be offered to everyone as they were receiving only generic feedback after summative OSCEs at the time. Some felt they will likely not use the feedback because they have already passed the exam. One student felt that one-on-one feedback they received after failing an exam was unsatisfactory as the feedback giver neither appeared to know nor care about anything apart from what was written on the mark sheet. Feedback intervention Using digital tablets in summative OSCEs to produce feedback was feasible, acceptable and efficient. Exams were delivered smoothly without any disruptive incidents. Quality Seventy examiners assessed 287 students in eight stations resulting in 2296 assessments. Nearly two thirds of these (N = 1600) produced free text feedback, of which four were illegible. No unprofessional or derogatory comments were found. More than half of the comments (N = 857) were descriptions of two or more actions performed by the student. A quarter of the free text feedback (N = 387) contained both a description what the student did and a suggestion for improvement. A minority contained only a general description or unspecific judgement (N = 240). Quantity Nearly all assessments (N = 2266) produced either free text or pre-written “tickbox” feedback. All students received either free text or tickbox feedback in all stations. All students received both types of feedback in at least two stations and nearly all students (N=271, 94.43%) did so in at least four stations out of eight. Quantity distribution of both free text and tickbox counts followed closely to normal distribution (mean = 125.73 and 32.45, SD = 50.88 and 5.37, skew = 0.67 and 0.12 respectively). Failed students received significantly more free text feedback than the rest (P = 0.00). Engagement A large majority (N=263, 91.64%) of students viewed their feedback after the main exam. Honours students’ view rate (92.59%) was higher than that of the passed students (89.74%). A smaller proportion of students viewed their feedback after the resit (78.95%). A smaller proportion of passed students at resit (N=26, 83.87%) viewed their feedback while only three out of seven failed students did so. FOS study Nearly a third of all students (N = 422) responded to the FOS questionnaire. All items had high reliability (SE = 0.03, coefficient alpha = 0.89). Confirmatory factor analysis produced good fit indices which supported the applicability of the model to the data. Overall feedback orientation was high (mean = 3.99). Utility domain had the highest mean at 4.3 and the Feedback self-efficacy domain had the lowest mean at 3.7. The overall scores were significantly higher in students from training years four and five (p = 0.02). Males had a significantly higher self-efficacy score than females (p = 0.02). GEM students had significantly higher scores in Utility and Social Awareness domains than non-GEMs (p = 0.02 and 0.03 respectively). The Utility scores increased in year four and five significantly (p = 0.02), while Social Awareness scores were significantly lower in the first year (p = 0.00). Females were significantly stronger in Utility than males when they are GEM students (p = 0.02). There was no other significant difference in any domain according to student variables. Discussion Feedback after summative OSCEs was successful: quality and quantity of feedback for all students was adequate and learner engagement was high. This success could be attributed to the information gained during the literature review and peer interviews. Examiner training should include specific information regarding what constitutes good feedback in order to further improve feedback quality. Finding a direct relationship between academic performance and feedback engagement was congruent with existing evidence. It is not surprising because it is logical to assume that higher performing students are more motivated learners who would engage with feedback, or any learning opportunity, more than lower performing or unmotivated learners would. Examiners showed a tendency to help poor performing learners correct their mistakes than help competent ones perfect their practice by giving significantly more feedback to poor performing students. This is likely because examiners felt it was important to help students pass and also to prevent incompetent practice. More evidence should be attained on the examiners’ feedback giving behaviour. This study generated strong psychometric evidence of feedback orientation in medical education using data from a large number of students. FOS was reliable to measure the same constructs in medical education as it did in business management. None of its items were redundant. Overall feedback orientation of the medical students was strong. It was not surprising to find more senior students and GEMs having higher feedback orientation and males having more Feedback self-efficacy than females. Students were aware of the utility of feedback more than they were confident about their ability to use the feedback they receive. Despite these variations, no demographic group appeared to be severely weak at any aspect of feedback orientation. Medical educators can use FOS to assess how receptive their learners are to feedback, both as a group and individually, and give feedback accordingly. Future research should focus on the relationship between the learners’ feedback orientation and their actual engagement with feedback, for example, in terms of whether they read the feedback they receive. Targeted interventions to improve each aspect of feedback orientation should be developed. In conclusion, this study generated evidence on providing feedback to undergraduate medical students after summative OSCEs and on their feedback orientation