Towards Automatic Visual Recognition of Horse Pain

Pain is a manifestation of disease and decreases welfare. Early detection of animal pain can not only improve animal well being by enabling early diagnosis and treatment of disease, but can also reduce healthcare costs for livestock owners. A video based animal pain detection system can provide a reliable and scalable means to unobtrusively monitor animals round the clock for signs of pain behavior, and enable the timely provision of medical treatment and pain management. This thesis presents the first steps towards creating such an automated visual system for animal pain detection. In particular, it presents computer vision techniques for pain recognition in the horse, and addresses the challenges of reliably determining pain when working with small scale and sparsely annotated datasets. We first present two methods that address challenges in veterinary research on equine pain detection by transferring techniques from computer vision and graph theory. We present a unifying description of the equine pain face by use of the biologically grounded language of Equine Facial Action Coding System (EquiFACS) to identify facial changes most correlated with pain in horses. In addition, we develop a novel graph based method that deduces the components of pain expression in horses by inspecting correlations between facial changes. Following, we develop an automatic and easy to use application for finding horse faces in videos that allows veterinary researchers to quickly identify time segments suitable for facial expression annotation from long videos. The application uses a deep convolution network for fast and reliable detection of horse faces, saves veterinary researchers hours in valuable annotation time, enables blinding during the data selection process, and has been instrumental in the description of both the pain and the stress face in horses in terms of EquiFACS. Beyond veterinary science, we present novel computer vision methods for automatic horse behavior understanding that use small and sparsely annotated datasets. We present a means for identifying facial keypoints in animal faces that enables accurate detection of horse facial parts without requiring large amounts of training data by transferring knowledge from large, readily available human facial keypoint datasets via face structure warping. Apart from facial keypoints, collecting detailed horse pain annotation in videos is cumbersome, and unscalable. We address this problem by developing two different methods that are capable of identifying pain behavior with crude -- weak -- video level training labels. First, we present a graph convolution based method for action localization that, by the explicit use of similarity relationships between time segments in videos, temporally localizes the extent of actions in videos despite not being trained with any such annotation. Finally, we present a method for pain detection in horses that uses horse pose cues, learned via multi-view surveillance footage, in a weakly supervised setting to deduce the pain status of the horse. The method identifies pain features that align well with pain scales currently used by veterinary practitioners, with impressive accuracy