Combined metrics for the evaluation of air traffic controller assistance systems

Today's assistance systems in aviation, as an arrival manager (AMAN), are able to plan efficient approach sequences and propose suggestions for optimized approaches. The AMAN acts as a suggestion system that proposes a plan and supports the air traffic control officer (ATCO) by implementing it. If the ATCO deviates from the instructions proposed by the AMAN, such as change of heading, altitude or speed, this leads to a changed flight profile. To update the planning and suggestion process current AMAN implementations use radar information to detect the changed flight profiles. The AMAN update thus depended on radio transmission, reaction time of the pilot and the implemented algorithms for change detection. Worst case, the updated planning between the pilot and sequencing of the AMAN could last more than 30 seconds. As part of the project AcListant® (Active Listening Assistant) two modalities were compared, using the ATCO behavior directly to distinguish changes to the flight path. A modality uses the mouse as input device for changes to the planning and the other uses the radio communication of the ATCO by analyzing their commands with Automatic Speech Recognition (ASR). This allows the AMAN to detect and implement changes to the planning by the ATCO much faster than before. The modalities were tested in a validation campaign with three operational experts in two different scenarios. The used traffic scenarios ensured a high degree of realism and an average traffic load. The metrics to evaluate the modalities can be categorized into safety (e.g. number of conflicts), efficiency (e.g. delay), and workload (NASA Task Load Index Score and Instantaneous Self Assessment). Also all communication of the participants were transcribed and analyzed to obtain the usage of the radio channel and also information about the context. Since the ATCO task is complex none of the selected metrics alone is suitable to give a statement about the quality of the modalities. An addition challenge is the fact that the AMAN and the modalities operate on a suggestion level and therefore is not binding for the ATCO. We chose a task analysis based on radio communication and observation to find a reasonable combination of different metrics and thus provide a comprehensive insight into the influence of modalities on the ATCO behavior. The combination of metrics is used to perform an assessment of the modalities. Therefore the assessment refers to both the complexity of the working environment as well as the observed ATCO activities