Sensing, Measuring, and Modeling Social Signals in Nonverbal Communication

Humans convey their thoughts, emotions, and intentions through a concert of social displays: voice, facial expressions, hand gestures, and body posture, collectively referred to as social signals. Despite advances in machine perception, machines are unable to discern the subtleand momentary nuances that carry so much of the information and context of human communication.The encoding of conveyed information by social signals, particularly in nonverbal communication, is still poorly understood, and thus it is unclear how to teach machines touse such social signals to make them collaborative partners rather than tools that we use. A major obstacle to scientific progress in this direction is the inability to sense and measure the broad spectrum of behavioral cues in groups of interacting individuals, which hinders applying computational methods to model and understand social signals.In this thesis, we explore new approaches in sensing, measuring, and modeling social signals to ultimately endow machines with the ability to interpret nonverbal communication. This thesis starts by describing our exploration in building a massively multiviewsensor system, the Panoptic Studio, that can capture a broad spectrum of human social signaling—including voice, social formations, facial expressions, hand gestures, and body postures—among groups of multiple people. Second, leveraging this system equipped with more than 500 synchronized cameras, we then present a method to measure the subtle 3D movements of anatomical keypoints in face-to-face interaction, providing a new opportunityto computationally study social signals. In the last part of this thesis, we present a social signal prediction task to model nonverbal communication in a data-driven manner. We establish a new large-scale corpus from hundreds of participants containing various channels of social signal measurements. Leveraging this dataset, we verify that the social signals are predictive each other with strong correlations