Depth-Aware Artistic Neural Style Transfer for Images, Video and Games

Neural Style Transfer (NST) is concerned with the artistic stylisation of various forms of data, such as images, videos and 3D models. Predominantly applied to images, NST involves transferring the style of an input artwork onto an input content image resulting in a stylised output that preserves the contents of the original image while embodying the artistic influences, such as texture and colour, of the reference artwork. The challenge in synthesising artistically stylised visual media lies in achieving content preservation and sufficient style fidelity, along with addressing temporal consistency for sequential data, and speed for real-time applications. This thesis explores the role of depth data, inferred using state-of-the-art depth estimation methods or provided as ground truth, in enhancing artistic style transfer for images, video, and 3D computer games. The research introduces four novel contributions.
First, a depth-aware image style transfer approach is presented that adopts a more accurate depth estimation network compared to previous approaches and replaces the batch normalisation layers in the convolutional neural network (CNN)-based architecture of the stylisation network with instance normalisation. It exhibits results of enhanced preservation of the global structure and depth information of the input image.
Second, the utilisation of depth information for enhanced video stylisation quality and improved temporal consistency is examined. A depth-aware video style transfer approach is presented that utilises encoded depth information and combines depth with optical flow data for a novel temporal loss function that is shown to produce stable stylisations for both synthetic and real-world video data.
The third contribution focuses on artistic style transfer for 3D computer games. For games, while memory and speed are core challenges, conventional 3D rendering pipelines provide additional intermediate information that can be exploited. A novel approach that integrates a fast depth-aware stylisation network into a computer graphics pipeline is presented. To leverage G-buffer data, such as depth and normals, that becomes available during the rendering process, an approach that utilises G-buffer information both during training and inference is also developed. In addition, the challenge of arbitrary in-game stylisation is investigated and a stylisation framework based on a novel perceptual quality-guided knowledge distillation scheme is presented. The in-game approaches utilise depth information, which as demonstrated for images and video, is found to be beneficial for enhancing the quality of stylised game worlds.
Lastly, the thesis provides an analysis of the current evaluation landscape in style transfer research, highlighting inconsistencies and discussing policies and principles for establishing a standardised evaluation procedure.