Beyond Automation Surprises: A Simulator Study of Disturbance Management on Highly Automated Flight Decks

Error prevention alone will never be sufficient for improving safety in complex high-risk systems, such as aviation. This approach needs to be combined with better support for error and disturbance management which, in turn, requires an improved understanding of current strategies for coping with errors and the resulting disturbances to the flight. The present research has sought systematic empirical evidence to expand our understanding of the disturbance management process on modern flight decks. A simulator study was conducted with twelve B747-400 airline pilots in order to examine (the effectiveness of) their strategies for diagnosing and recovering from disturbances, and the impact of current automation design on these processes. Pilots flew a one-hour scenario (with a confederate copilot) which contained challenging events that probed pilots’ knowledge of, and proficiency in, using the autoflight system. A process tracing methodology was used to analyze and identify patterns in strategies across pilots. Overall, pilots completed the scenario successfully but varied considerably in how they coped with disturbances to their flight path. Our results show that aspects of feedback design delayed the detection, and thus escalated the severity, of a disturbance. Diagnostic episodes were rare due to pilots’ knowledge gaps as well as timecriticality. Our findings can inform the development of design and training solutions to observed difficulties with error and disturbance management in a variety of domains