A Novel Efficient Deep Learning Framework for Facial Inpainting - Face reconstruction from masked images

The use of face masks due to the covid-19 pandemic has made surveillance of people very difficult. Since a mask covers most of the facial components, security cameras are rendered of little to no use in the identification of criminals. In order to realize what a face looks like behind a mask, we have to construct the facial features in the masked region. On a higher level, this falls under the field of image inpainting, i.e. filling missing regions of images or correcting irregularities in images. Current research on image inpainting shows promising results on images that have missing/incorrect patches or have manually blackened/whitened pixels. However, they have not been designed to handle the specific case of facial feature generation beneath a mask. The challenge lies in the fact that the mask becomes a bunch of pixels that are on the face in different natural colours, types(surgical, cloth, etc) and shapes(N95, N90, etc). In this paper I provide a custom U-Net based Convolutional Neural Network to regenerate the face underneath the mask. I also create a synthetic dataset of artificially masked human faces with various types of masks. Initial experiments demonstrated superior performance in terms of both quality, diversity over different types of facial coverings/masks and free-form image completion with little to no error