Monitoring pilot’s cognitive fatigue with engagement features in simulated and actual flight conditions using an hybrid fNIRS-EEG passive BCI

There is growing interest for implementing tools to monitor cognitive performance in naturalistic environments. Recent technological progress has allowed the development of new generations of brain imaging systems such as dry electrodes electroencephalography (EEG) and functional near infrared spec- troscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. These highly portable brain imaging devices offer interesting prospects to implement passive brain computer interfaces (pBCI) and neuroadaptive technology. We developed a fNIRS-EEG based pBCI to monitor cognitive fatigue using engagement related features (EEG engagement ratio and wavelet coherence fNIRS based metrics). This mental state is known to impair cognitive performance and can jeopardize flight safety. In this preliminary study, four participants were asked to perform four identical traffic patterns along with a secondary auditory task in a flight simulator and in an actual light aircraft. The two first traffic patterns were considered as the low cognitive fatigue class, whereas the two last traffic patterns were considered as the high cognitive fatigue class. As expected, the pilots missed more auditory targets in the second part than in the first part of the experiment. Classification accuracy reached 87.2% in the flight simulator condition and 87.6% in the actual flight conditions when combining the two modalities. This study demonstrates that fNIRS and EEG-based pBCIs can monitor mental states in operational and noisy environments