The Development and Evaluation of a Behaviorally Based Rating form for the Assessment of En Route Air Traffic Controller Performance

This project expanded and evaluated the performance evaluation method developed by Sollenberger, Stein, and Gromelski (1997), a Terminal Radar Approach Control rating form and training package designed to better assess air traffic controller performance. The form is a research-oriented testing and assessment tool designed to measure the efficacy of new air traffic control (ATC) systems, system enhancements, and operational procedures in simulation research. The rating form used in the present study focused on observable behaviors that supervisory air traffic control specialists (SATCSs) use to make behaviorally based ratings of en route controller performance. The present study evaluated the inter-rater and intra-rater reliability of performance ratings made by nine Air Route Traffic Control Center supervisors who viewed videotapes and computerized replays of controllers from a previously recorded en route study. The rating form contained 26 items, which were organized into six major categories. Various observable behaviors, which SATCSs identified as those they consider when assessing controller performance, anchored each performance area. Inter-rater (between rater) reliability of SATCS performance ratings assessed using intra-class correlations was somewhat low. Intra-rater (within rater) reliability of SATCS performance ratings was consistent with previous studies and indicated that raters were stable over time in the ratings they assigned. Researchers also investigated the relationship between SATCS performance ratings and personality traits from the Sixteen Personality Factor personality inventory. The results indicated that what SATCSs bring with them to the experimental evaluation setting, in terms of personality traits, may be related to their ratings. Future research efforts should concentrate on distinguishing the sources of measurement error and making whatever changes necessary to produce a reliable controller performance assessment tool