Establishing benchmarks for minimum competence with dry lab robotic surgery drills.

The robotic objective structured assessment of technical skills (R-OSATS) is a previously validated assessment tool that is used to assess five standardized inanimate robotic surgery drills. R-OSATS is used to evaluate performance on surgical drills, with scores of 0-20 for each drill. Our objective was to establish the minimum threshold score that denotes competence on these drills. Thus, we performed a standard-setting study using data from surgeons and trainees in eight academic medical centers. Cutoff scores for the minimal level of competence using R-OSATS were established using two techniques: 1) the Modified Angoff; and 2) the Contrasting Groups methods. For the Modified Angoff method, eight content experts met, and in an iterative process, derived the scores that a minimally competent trainee should receive. After two iterative rounds of scoring and discussion with the Modified Angoff method, we established a minimum competence score per drill with high agreement (rWG range 0.92 - 0.98). There was unanimous consensus that a trainee needs to achieve competence on each independent drill. A second method, the Contrasting Groups method, was used to verify our results. In this method, we compared R-OSATS scores from inexperienced (34 PGY-1 and 2 trainees) versus experienced (22 faculty and fellow) robotic surgeons. The distributions of scores from both groups were plotted and a cutoff score for each drill was determined from the intersection of the two curves. Using this method, the minimum score for competence would be 14 per drill, which is slightly more stringent but confirms the results obtained from the Modified Angoff approach. In conclusion, using two well described standard-setting techniques, we have established minimum benchmarks designating trainee competence for five dry-lab robotic surgery drills