Model-based Self-supervision for Dense Face Alignment and 3D Reconstruction

In the field of monocular 3D reconstruction, self-supervision based on differentiable rendering and a statistical 3D model has been proposed to alleviate the need for datasets with ground truth. In theory, this enables training of neural networks only using unannotated images. However, training through self-supervision tends to be unstable and surrogate supervision such as landmarks is required in practice. Moreover, reaching convergence in self-supervised 3D reconstruction is slow or unachievable due to the weak and discontinuous supervisory signal provided by a differentiable renderer. Our research starts from the aim to improve such problems in differentiable renderer-based self-supervision.

Firstly, we combined differentiable linear least-squares fitting of a 3D morphable model (3DMM), pose, and lighting with self-supervision. We propose linear least-squares solutions for geometric and photometric parameters including a novel inverse spherical harmonic lighting model. This assures optimal fitting of photometric components given estimated geometric parameters and improves fidelity in reconstructed appearance. This concept also provides an opportunity to combine 3DMM fitting with image-to-image networks, leading to stable training without requiring landmark supervision.

Secondly, we proposed supervision based on semantic segmentation. In contrast to landmarks, this form of supervision is dense and always well defined. However, it is not one-to-one, meaning more complex loss functions are required to exploit it. We propose two novel cohesive measures for semantic segmentation supervision. First, we show how precomputed distance maps in a 3DMM UV space can be used to supervise pixel-wise estimates of image-model correspondence. Second, we derive a novel differentiable vertex to pixel cohesive measure based on the geometric Renyi divergence. Using this loss, we show that pure shape-from-semantic segmentation is possible via analysis-by-synthesis.

Lastly, we combined both techniques and propose the self-supervised architecture for 3D face reconstruction that does not require a differentiable renderer.