Effective and Efficient Hand Detection in Egocentric Videos

Background. Individuals with spinal cord injury (SCI) report upper limb function as their top recovery priority. To accurately represent the true impact of new interventions on patient function, evaluation should occur from home. Recent work has used wearable cameras to monitor hand function of individuals with SCI in a home environment, and these egocentric videos were automatically analyzed using computer vision. A key step of this process, hand detection, is difficult to accomplish robustly and reliably, hindering the deployment of a complete monitoring system in the home. Objective. Generate an algorithm for fast and reliable hand detection in egocentric videos by combining object detection and tracking algorithms. Methods. We generated an egocentric hand detection dataset (167,622 frames) using videos of 17 individuals with SCI performing activities of daily living in a home simulation laboratory. Using existing detection and tracking algorithms, we introduced two novel combination methods: 1) using detectors to reset trackers and 2) using trackers to generate hand proposals for classification. Results. Method 1 resulted in an algorithm that was twice as fast as the fastest detector alone while being twice as accurate as the best tracker alone. Method 2 resulted in the fastest proposal generation method for hand detection with competitive recall on 1 public dataset. Conclusion. Robust and reliable hand detection that is efficient on portable CPUs will help clinicians directly measure hand function in a patient’s daily life at home.M.A.S