Finding Group Interactions in Social Clutter

We consider the problem of finding distinctive social in-teractions involving groups of agents embedded in larger social gatherings. Given a pre-defined gallery of short ex-emplar interaction videos, and a long input video of a large gathering (with approximately-tracked agents), we identify within the gathering small sub-groups of agents exhibiting social interactions that resemble those in the exemplars. The participants of each detected group interaction are lo-calized in space; the extent of their interaction is localized in time; and when the gallery of exemplars is annotated with group-interaction categories, each detected interaction is classified into one of the pre-defined categories. Our ap-proach represents group behaviors by dichotomous collec-tions of descriptors for (a) individual actions, and (b) pair-wise interactions; and it includes efficient algorithms for optimally distinguishing participants from by-standers in every temporal unit and for temporally localizing the extent of the group interaction. Most importantly, the method is generic and can be applied whenever numerous interacting agents can be approximately tracked over time. We evalu-ate the approach using three different video collections, two that involve humans and one that involves mice. 1