Reconstruction meets Recognition: Exploring Face Identity Embeddings

Embedding a face image into a descriptor vector using a deep neural network is a standard technique in face recognition. These embeddings are designed to capture only identity (ID) information. Their effectiveness is assessed by evaluating recognition performance against datasets with diverse non-identity (non-ID) factors. This thesis examines the data within face embeddings and whether faces can be reconstructed from these embeddings using a Generative Adversarial Network (GAN) or a 3D Morphable Model (3DMM).

The first contribution of this work involves studying the ID and non-ID information encoded within face embeddings. Ideally, environmental data like background and lighting, along with variable facial aspects such as pose and accessories, should be ignored in recognition. However, we find that attributes, landmark positions, and image histograms can be retrieved from ID embeddings of networks like VGGFace2 and ArcFace. Building on this, we propose an adversarial training method that more effectively excludes non ID information by deploying a novel network architecture that selectively penalizes non-ID attribute encoding, enhancing recognition performance.

The second contribution focuses on reconstructing 2D facial images directly from embeddings. Our study reveals that these reconstructed images contain not only identity but also certain non-ID characteristics such as pose, lighting, and background. These findings highlight the privacy risks inherent in facial recognition technologies.

The final contribution shifts focus to the potential reconstruction of 3D face geometry solely from recognition signals. By integrating a 3DMM with a Spatial Transformer Network (STN), we demonstrate that the localiser network can learn 3D shape and pose parameters from identity signals in a warped UV space without additional geometric labels. Our findings confirm that face representations retain spatial information for 3D geometry. This potentially points to future face recognition architectures where most of the model capacity is used for alignment, leaving a relatively simple feature extraction and recognition problem on the aligned face image.