Physiological Assessment of Engagement during HRI: Impact of Manual vs Automatic Mode

The employment of physiological measurements to perform on-line assessment of operators' mental states is crucial in the field of human-robot interaction (HRI) researches and is still an open topic to the best of our knowledge. In order to progress towards systems that would dynamically adapt to operators' mental states, a first step is to determine an adequate protocol that elicits variations in engagement. To this purpose, this work focuses on analyzing operator's physiological data streams recorded during a human-robot mission executed in a original virtual environment [Drougard17]. In detail, the conceived mission consists in a mutual cooperation of a firefighter robot and its human operator to extinguish fires. A high level of complexity is obtained by the number and the random nature of events to be handled during the mission. As an example, guiding the robot, managing its water tank level and taking care of its electric charge are all tasks to be accomplished simultaneously, which can randomly be assigned to autonomous or manually operative mode. In addition to that, an extra task consisting in keeping an adequate level of an external water tank to allow robot refill is assigned only to the human operator for continuously soliciting his attention. Anyone can experience this mission by visiting the website robot-isae.isae.fr set up to collect a large amount of behavioral data by crowdsourcing. The mission is accomplished through a remote human-machine interface made of controllers and a screen displaying different areas corresponding to each task. Figure 1 shows the graphical user interface, with the 5 areas of interest (AOIs). The control station is equipped with sensing devices for human data collecting. The sensing devices which allows for human data collection are an eye-tracker (SMI), located on the bottom bar of the display and a portable bluetooth electrocardiograph (eMotion Faros 360). A specific procedure for the experiment has been defined to guarantee a statistical significance of the recorded dataset. In detail, each different human operator has to complete at least three times a ten minutes mission aiming to the best score in terms of fires extinguished. An absolute resting is imposed between missions to get a baseline for the cardiac activity. The data are collected on 17 participants of mixed sex (9 females) with average age 28.5 (S.D. = 4.52). The number and the duration of fixations per area of interest are extracted from the eye-tracker. The length of inter-beat intervals, the Heart Rate Variability (HRV) and instant Heart Rate Variability (IHRV) are computed from the ECG. Preliminary results show a lower HRV and IHRV during the mission than during the rest session: Student and Wilcoxon statistical tests ensure a difference at least equal to 6 (p<0.05). This evidence, according to the literature, highlights that the created mission succeeded in engaging the participants. The impact of each mode of operation (manual/autonomous) on the human markers is also observed and analyzed. Contrarily to expectations the operator results more engaged (lower HRV) during autonomous than manual mode (p<0.05). This is in accordance with the tasks' difficulty. In fact, when the autonomous mode takes over, the human priority is the only task that he has to accomplish by himself (external tank filling), which is also designed to be the hardest one. This is confirmed by IHRV that in average is greater during manual mode. Moreover, finding that HRV and IHRV behaving in the same way, a real-time Human-Robot team supervision application can be foreseen. The effect of the current mode of operation is observable also on the number and durations of fixations on the two main AOIs: video streamed from the robot and external water tank level. The first attracts the operator attention mainly when in manual mode, the second when in autonomous mode (p<0.05). Spearman correlations of data sample per second confirm previous results. Indeed, markers on these AOIs are correlated with the mode of the robot (rho=0.22, p<0.05) as well as IHRV (rho=0.03, p<0.05). Several kind of correlation have been identified analyzing the recorded dataset, of which the most significant to describe the human operator engagement are found to be: the number and durations of fixations on AOIs corresponding to the two main tasks are negatively correlated (rho=-0.6, p<0.05) which describes the fact that the human tends to switch attention mainly between these two tasks; the correlation of IHRV marker to the markers on the AOIs (tank: rho=0.1, p<0.05; video: rho=-0.06, p<0.05) demonstrates that the main tasks are perceived as so from the human operator, while its correlation to the remaining mission time (rho=0.05, p<0.05) expresses a higher engagement as the mission progresses. Moreover, IHRV is also negatively correlated to performance indexes as the number of extinguished fires (rho=-0.07, p<0.05) and the external tank level (rho=-0.14, p