The Development and Evaluation of a Behaviorally Based Rating Form for Assessing Air Traffic Controller Performance

This document DOT/FAA/CT-TN96/16, dated May 1997 supersedes DOT/FAA/CT-TN96/16 dated February 1997. The evaluation of air traffic controller performance is a complex process. While there are standard forms in field use, there is currently no comprehensive system for reliable observer evaluation. This research involves the development of a new form along with a training package for use in research and possibly operational testing. The form consists of 24 rating scales. These scales focus on observable actions that trained air traffic control specialists could identify to make behaviorally based ratings. The study evaluates the reliability of the rating form by determining the consistency of ratings obtained from six observers who viewed videotapes of a previously recorded simulation study. These observers were supervisors and training staff specialists from Terminal Radar Approach Control facilities nationwide. Prior to making formal ratings, the observers participated in a training program designed to help them become proficient in observational rating. During the evaluation phase of the study, the observers viewed 20 one-hour videotapes of controllers working different traffic scenarios. The results indicated that most of the rating scales had reasonable inter-rater reliabilities ranging from r = .7 to r = .9. The study also identified the performance areas that were more difficult for observers to evaluate consistently, possibly due to misunderstanding rating criteria or overlooking critical controller actions