Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions

Hands appear very often in egocentric video, and their appearance and pose give important cues about what peo-ple are doing and what they are paying attention to. But existing work in hand detection has made strong assump-tions that work well in only simple scenarios, such as with limited interaction with other people or in lab settings. We develop methods to locate and distinguish between hands in egocentric video using strong appearance models with Con-volutional Neural Networks, and introduce a simple can-didate region generation approach that outperforms exist-ing techniques at a fraction of the computational cost. We show how these high-quality bounding boxes can be used to create accurate pixelwise hand regions, and as an applica-tion, we investigate the extent to which hand segmentation alone can distinguish between different activities. We eval-uate these techniques on a new dataset of 48 first-person videos of people interacting in realistic environments, with pixel-level ground truth for over 15,000 hand instances