Audiovisual detection of behavioural mimicry

Abstract—Human mimicry is a behavioural cue occurring dur-ing social interaction that can inform us about the participants’ inter-personal states and attitudes. It occurs when a participant in an interaction exhibits some behaviour as a result of a co-participants prior display of that signal, and occurs on both short and long time-scales. To develop a detection method for such behaviour, we use a method based on feature prediction, where we train an ensemble of regression models from one subject’s features to the co-subject’s features, for each class. The ensemble of models with lowest reconstruction error is used to detect mimicry and non-mimicry, using continuous audiovisual streams. As mimicry events are dynamical phenomena, we use a temporal regression model (long short-term memory neural networks) to capture sequential dependencies in the data. On a data set of ten 12-minute dyadic interaction episodes, our method gave average positive and negative recall rates of 77.5 % and 60.0 % respectively, on data with significant class imbalances, due to the relative sparsity of mimicry samples when doing continuous detection. I