Neural field multi-view shape-from-polarisation

In this thesis, we provide three novel contributions towards 3D reconstruction by leveraging polarimetric information. First, we modify NeRF to work with the input obtained from a polarisation camera. In particular, we extend NeRF to cover 12 channels of the camera sensor. Unlike previous works, this is the first time that the model is fitted directly to raw polarisation sensor data, bypassing the need for demosaicing. Since the polarisation state of reflected light encodes the surface normal used for reconstructing 3D geometry, our method provides richer information about surface orientation than RawNeRF which uses conventional raw RGB images. This form of input is challenging for the model training due to input sparsity. Nonetheless, we show that this setup works reasonably well with a synthetic dataset, while requiring additional constraints for real-world capture. Secondly, we link surface geometry with polarised radiance through a mixed polarisation model and then inject the physical insights into the training pipeline - significantly improving the geometry prediction of the object in the scene. Rather than guessing the relationship between captured data and surface orientation (as in a 12-channel black box model), the physics-based model could follow the physical rule given by the mixed polarisation model. Nevertheless, despite its physical understanding, this model neglects practical limitations. Therefore, our last contribution is to investigate the reasons why the model did not behave as expected and tackle the issues related to noise and saturation, which greatly improve the quality of 3D reconstruction - achieving state-of-the-art performance on the PANDORA benchmark.