Face reconstruction and reenactment have been extensive research for decades due to their fundamental research status and wide applications. Previous reconstruction methods request multi-view input or different illumination conditions. Current applications require less constrained settings, even from a monocular image. These approaches rely on face priors, commonly known as 3D Morphable Models (3DMMs). One target of this thesis aims to use the analysis-by-synthesis idea to reconstruct 3DMMs from monocular input. We propose a self-supervised training architecture considering the 3DMM as a face prior. We analyze the 3D face by Deep Neural Network (DNN) and then synthesize the 2D facial landmark and rendering results by projection and differentiable rendering. Then, we leverage the synthes...[ Read more ]

Face reconstruction and reenactment have been extensive research for decades due to their fundamental research status and wide applications. Previous reconstruction methods request multi-view input or different illumination conditions. Current applications require less constrained settings, even from a monocular image. These approaches rely on face priors, commonly known as 3D Morphable Models (3DMMs). One target of this thesis aims to use the analysis-by-synthesis idea to reconstruct 3DMMs from monocular input. We propose a self-supervised training architecture considering the 3DMM as a face prior. We analyze the 3D face by Deep Neural Network (DNN) and then synthesize the 2D facial landmark and rendering results by projection and differentiable rendering. Then, we leverage the synthesis results to provide supervision for the DNN training. However, the 3DMM is a statistic model learned from datasets of 3D scans, which contains limited expressions and loses face detail. Hence, another goal of this thesis is to construct a digital avatar construction pipeline, which provides avatars with vivid face shapes and expressions to build a face model. Besides, it can also offer face details. The last goal of this thesis is 3D-awareness face reenactment, which synthesizes images combining the source image identity and the facial expression and pose of driving video frames under 3D consistency. Recent methods regard 2D landmarks as face shape, expression, and pose representations, which are affected by the entangling of face shape, expression and pose. Hence, we push the state-of-the-art in high-quality face reenactment in a synthesis-by-analysis strategy. In this way, we synthesize images under the analysis of the 3D face, which is a more robust representation than facial landmarks. Hence, this thesis’s key idea is to analyze and synthesize human faces. Finally, we construct a facial expression recognition network, where face analysis and synthesis are applied to improve the recognition results.