Reprints and permission:

Sponsorships or competing interests that may be relevant to content are dis-closed at the end of this article. Objective. To determine the interrater reliability and con-struct validity of 3 separate assessment tools for assessing trainee skills in pediatric airway endoscopy simulation. Design. An Objective Structured Assessment of Technical Skills (OSATS) was developed in which examinees were asked to name and assemble the airway foreign body instruments and retrieve a for-eign body from an infant airway mannequin. Each examinee’s perfor-mance was assessed in a blinded fashion by 3 pediatric otolaryngology faculty at separate residency programs using 3 assess-ment tools: (1) objective quantifiable measures list (eg, assists needed, forceps openings, foreign body drops), (2) 15-point OSATS checklist, and (3) Global Rating Index for Technical Skills (GRITS). Setting. Otolaryngology residency program. Subjects. Examinees (medical students, n = 3; otolaryngology residents, n = 17; pediatric otolaryngology faculty, n = 3) and raters (n = 3). Main Outcome Measures. Interrater reliability and construct validity. Results. Anonymized split-screen videos simultaneously capturing each examinee’s instrument handling and the endoscopic videos were created for all 23 examinees. Nineteen videos were chosen for review by 3 raters. The interrater reliability as mea-sured by the intraclass correlation for objective quantifiable mea-sures ranged from 0.46 to 0.98. The intraclass correlation coefficient was 0.95 for the 15-point OSATS checklist and 0.95 for the GRITS; both showed a high degree of construct validity with scores correlating with previous experience. Conclusion. Assessment tools for skills assessments must have high interrater reliability and construct validity. When assessing trainee skills in pediatric airway foreign body sce-narios, the 15-point OSATS checklist developed by this group or the GRITS meets these criteria