Hand segmentation with structured convolutional learning

Abstract. The availability of cheap and effective depth sensors has re-sulted in recent advances in human pose estimation and tracking. De-tailed estimation of hand pose, however, remains a challenge since fingers are often occluded and may only represent just a few pixels. Moreover, labelled data is difficult to obtain. We propose a deep learning based-approach for hand pose estimation, targeting gesture recognition, that requires very little labelled data. It leverages both unlabeled data and synthetic data from renderings. The key to making it work is to integrate structural information not into the model architecture, which would slow down inference, but into the training objective. We show that adding un-labelled real-world samples significantly improves results compared to a purely supervised setting.