[[alternative]]Occluded Face Recovery and Face Hallucination---Theory and Practice

計畫編號：NSC101-2218-E032-005 研究期間：201208~201307 研究經費：500,000[[abstract]]在許多的場合裡，我們常需要在遮蔽情況下或是解析不足的情況下辨識人臉影像。 典型的例子包括辨識監視系統裡解析度不足的罪犯影像，在這種情況下，因為人臉的主 要特徵被遮蔽住了，而使得識別變為困難。因此，我們提出了一個新的架構來自動化地 恢復影像中的遮蔽區域與重建高解析人臉影像。我們提出的計畫包含了三個連續的步 驟，即，1)從低品質輸入（具遮蔽情況或低解析度）的人臉影像中，找出人臉特徵點 的位置；2)人臉影像的遮蔽重建；3)將低解析的人臉影像重建成高解析影像。整個計 畫中，我們主要是利用direct combined model (DCM)方法來推斷低品質輸入影像裡所缺 乏的高品質部分。 自動化地重建人臉影像是極具挑戰性的工作，因為人臉影像通常具有多樣的姿勢， 表情和不同的尺寸，並有不同的光照和/或遮蔽的程度。在這個計畫裡，我們利用了所提 出的三個連續步驟來展示DCM的效能。首先，我們推導出一個貝氏框架把人臉重建步 驟、人臉校正步驟、以及遮蔽區域偵測步驟結合。針對這樣複雜的目標函數，我們打算 以粒子集（particle set)的方式來表示函數的分布情形。為了要快速且穩健地求解，一個 新的DCM方法，DCM-based particle-filter，被提出來。即使在不事先將遮蔽區域與影像 解析度恢復的情況下，DCM-based particle-filter的形式仍可正確地定位出人臉特徵點位 置。 在人臉遮蔽區域重建的步驟裡，我們將DCM演算法與Markov Random Field (MRF) 架構結合。所產生的組合：DCM-based MRF使得重建結果能同時考慮人臉的整體架構與 細微特徵。此外，針對這種MRF架構，我們也開發了一種有效的解法，DCM-based nonparametric belief propagation以得到更逼真的重建結果。關於第三個步驟：高解析人臉 重建，我們提出了 2維DCM和Kernel DCM兩種方法。2D DCM/Kernel DCM的主要想法 是，它們將影像以二維矩陣的方式在Kernel空間下呈現，而標準的DCM方法則是以一 維向量的方式在原始空間呈現。藉由這個概念，我們開發了一個最佳的方法得以找出高 解析人臉影像與低解析人臉影像在2D與Kernel裡的關係式。我們強調由於利用2D和 kernel的表達方法，人臉的全域與局部架構都能被保留，進而使得DCM轉換更有效率。[[abstract]]In numerous occasions there is need to identify subjects shown in heavily occluded face images or in low resolution images. Typical examples include the recognition of criminals whose facial images are captured by surveillance cameras that usually have a very low resolution. In such cases a significant part of the subjects face is occluded (or missing) making the process of identification extremely difficult. Consequently, we propose a new framework for automatic recovery of occluded face and automatic reconstruction of high-resolution face image. The proposed system executes three cascaded procedures, namely, 1) reconstruction of the facial shape from the input low-quality (occluded or low-resolution) facial image; 2) recovery of the occluded face; and 3) reconstruction of a high-resolution face image from an input low-resolution image, i.e. face hallucination. Our theoretical contribution is the direct combined model (DCM) approach for inferring a missing high-quality image from the low-quality input. Automatically reconstructing facial image is highly challenging because facial images typically exhibit a wide range of poses, expressions and scales, and have differing degrees of illumination and/or occlusion. We demonstrate the effectiveness of DCM approach in the proposed three procedures. First, we derive a Bayesian framework to unify the reconstruction stage with face alignment and occlusion detection, where the complex distribution of the recovery objective function is represented by a particle set. To perform such particle filter solution efficiently and robustly, a novel DCM method, DCM-based particle-filter, which utilizes the face specific prior knowledge is proposed. Such approach can accurately locate the facial shape without the need to restore the texture information lost as a result of unfavorable occlusion or resolution conditions. The occluded face recovery procedure draws together DCM and Markov Random Field (MRF). The resulting combination, DCM-based MRF, suggests a solution to integrate both the global and local face properties during the recovery process. Under such MRF structure, we further develop an efficient method, DCM-based nonparametric belief propagation algorithm, to create more photorealistic result. With regards to the third procedure, face hallucination, we present the 2-dimensional DCM (2D DCM) and Kernel-DCM methods. The main idea behind 2D DCM/kernel DCM is that it is based on 2D matrix in the Kernel space as opposed to the standard DCM, which is based on 1D vector in the original image space. We develop an optimization framework that finds an optimal transform to explain the 2D and Kernel relationships between the high-resolution face images and their smoothed and down-sampled lower resolution ones. We emphasize it is the 2D/kernel imagerepresentation that captures global and local face properties and makes the use of DCM transform more powerful.[[sponsorship]]行政院國家科學委員