On-the-Fly Hand Detection Training with Application in Egocentric Action Recognition

We propose a novel approach to segment hand regions in egocentric video that requires no manual labeling of train-ing samples. The user wearing a head-mounted camera is prompted to perform a simple gesture during an initial cal-ibration step. A combination of color and motion analy-sis that exploits knowledge of the expected gesture is ap-plied on the calibration video frames to automatically la-bel hand pixels in an unsupervised fashion. The hand pix-els identified in this manner are used to train a statistical-model-based hand detector. Superpixel region growing is used to perform segmentation refinement and improve ro-bustness to noise. Experiments show that our hand detec-tion technique based on the proposed on-the-fly training ap-proach significantly outperforms state-of-the-art techniques with respect to accuracy and robustness on a variety of chal-lenging videos. This is due primarily to the fact that train-ing samples are personalized to a specific user and envi-ronmental conditions. We also demonstrate the utility of our hand detection technique to inform an adaptive video sampling strategy that improves both computational speed and accuracy of egocentric action recognition algorithms. Finally, we offer an egocentric video dataset of an insulin self-injection procedure with action labels and hand masks that can serve towards future research on both hand detec-tion and egocentric action recognition. 1