Landmark detection for faces in the wild

2018-05-14Facial landmark detection has received much attention in recent years together with two detection paradigms emerging: local approaches, where each facial landmark is modeled individually and with the help of a shape model; and holistic approaches, where the face appearance and shape are modeled jointly. In recent years, both of these approaches have shown great performance gains for facial landmark detection even under real-world conditions (referred to as “in-the-wild”) of varying illumination, occlusion and image quality. However, their accuracy and robustness are very often reduced for profile faces where face alignment is more challenging (e.g., no more facial symmetry, less defined features and more variable background). Meanwhile, a key to successful face recognition is an accurate and reliable face alignment using automatically-detected facial landmarks. Given this strong dependency between face recognition and facial landmark detection, a robust face recognition requires knowledge of when the facial landmark detection algorithm succeeds and when it fails. Facial landmark confidence represents this measure of success. ❧ In this dissertation, we present the Holistically Constrained Local Model (HCLM) which unifies local and holistic facial landmark detection by integrating head pose estimation, sparse-holistic landmark detection and dense-local landmark detection. HCLM shows state-of-the-art performance, especially with the extreme head poses. We also present two methods to measure landmark detection confidence: local confidence based on local predictors of each facial landmark, and global confidence based on a 3D rendered face model. A score fusion approach is also introduced to integrate these two confidences effectively. It shows a significant accuracy improvement when face recognition algorithm integrates the local-global confidence metrics. ❧ Lastly, we extend the face recognition domain from still-images to videos. Videos, unlike still-images, offer a myriad of data for face modeling, sampling, and recognition, but, on the other hand, contain low-quality frames and motion blur. A key component in video-based face recognition is the way in which faces are associated through the video sequence before being used for recognition. We present a video-based face recognition method taking advantage of face and body association (FBA). To track and associate subjects that appear across frames in multiple shots, we solve a data association problem using both face and body appearance. The final recovered track is then used to build a face representation for recognition. Our end-to-end system shows state-of-the-art performances on the latest challenging video-based face recognition benchmark