A data-driven approach for facial expression synthesis in video

Abstract—This paper presents a data-driven approach for facial expression retargeting in video, i.e., synthesizing a face video of a target subject that mimics the expressions of a source subject in the input video. Our approach takes advantage of a pre-existing facial expression database of the target subject to achieve realistic synthesis. First, for each frame of the input video, a new facial ex-pression similarity metric is proposed for querying the expression database of the target person to select multiple candidate images that are most similar to the input. The similarity metric is devel-oped using a metric learning approach to reliably handle appear-ance difference between different subjects. Secondly, we employ an optimization approach to choose the best candidate image for each frame, resulting in a retrieved sequence that is temporally coherent. Finally, a spatio-temporal expression mapping method is employed to further improve the synthesized sequence. Experi-mental results show that our system is capable of generating high quality facial expression videos that match well with the input se-quences, evenwhen the source and target subjects have big identity difference. In addition, extensive evaluations demonstrate the high accuracy of the learned expression similarity metric and the effec-tiveness of our retrieval strategy. Index Terms—Facial expression, expression retargeting, expres-sion synthesis, expression similarity metric, data-driven. I