Facial Performance Synthesis Using Deformation-Driven Polynomial Displacement Maps

markers neutral mesh frequency displacements frequency displacements geometry Figure 1: We synthesize new high-resolution geometry and surface detail from sparse motion capture markers using deformation-driven polynomial displacement maps; our results agree well with high-resolution ground truth geometry of dynamic facial performances. We present a novel method for acquisition, modeling, compression, and synthesis of realistic facial deformations using polynomial dis-placement maps. Our method consists of an analysis phase where the relationship between motion capture markers and detailed fa-cial geometry is inferred, and a synthesis phase where novel de-tailed animated facial geometry is driven solely by a sparse set of motion capture markers. For analysis, we record the actor wearing facial markers while performing a set of training expression clips. We capture real-time high-resolution facial deformations, including dynamic wrinkle and pore detail, using interleaved structured light 3D scanning and photometric stereo. Next, we compute displace-ments between a neutral mesh driven by the motion capture mark-ers and the high-resolution captured expressions. These geometric displacements are stored in a polynomial displacement map which is parameterized according to the local deformations of the motion capture dots. For synthesis, we drive the polynomial displacement map with new motion capture data. This allows the recreation of large-scale muscle deformation, medium and fine wrinkles, and dy-namic skin pore detail. Applications include the compression of existing performance data and the synthesis of new performances. Our technique is independent of the underlying geometry capture system and can be used to automatically generate high-frequency wrinkle and pore details on top of many existing facial animation systems.