Capturing human behaviour through wearables by computational analysis of social dynamics

Understanding human behaviour has sparked the minds of many throughout centuries. One intriguing aspect of human behaviour is the social part; how humans react to each other and their environment. Scientifically studying such behaviour is hampered because of the need for manual annotations, so that social scientists limited themselves to observing only short time intervals in limited settings. With the growing processing power of computers and increasing possibilities of robust, continuous, and mobile sensing, collecting and analysing large amounts of real-life behaviour data has become possible. Moreover, computational methods make it possible to go beyond traditional approaches for social understanding, since they detect patterns that are not easily distinguishable for humans. However, even with powerful computational models, investigating human behaviour is quite challenging as behaviour is personal and contextual, resulting in huge variations. This thesis proposes novel computational solutions for analysing human social behaviour. It focusses on data collected from people with wearable accelerometers in crowded events where people freely mingle with each other. It provides solutions to robustly detect actions and interactions, as well as how to use the detected information to derive higher level social understanding. The thesis starts by introducing novel ways of detecting social actions and interactions. To deal with intra personal variations, we show how general action predictors can be adapted to become personalized models using the transfer learning methodology. Further, we show that the detection of conversing groups can be deduced from interaction dynamics, instead of the mainly preferred modality of proximity. Large variations of interaction patterns that might arise in unrestricted scenarios are addressed by a novel method that considers the sizes of the groups; both in training and detection phases. The thesis continues with a proof-of-concept study that shows how detected action and interaction patterns of people can be used to infer an individuals’ psychological construct. We show that it is possible to detect the construct of personality in a real life event by imitating two behavioural cues (speaking and movement) from one digital modality (acceleration). Additionally, we describe a detailed investigation of how social context moderates an individuals’ evaluation of a live performance. Through a novel approach, we infer audience members’ evaluations from informative parts of the event, identified by the linkage of body accelerations. Taken together, with this thesis we show that with the increased sensing and computing power, the understanding of human social behaviour in more dynamic social situations is within reach