White Rose University Consortium logo
University of Leeds logo University of Sheffield logo York University logo

Deep Complex-Valued Neural Networks for Natural Language Processing

Mönning, Nils (2019) Deep Complex-Valued Neural Networks for Natural Language Processing. PhD thesis, University of York.

[img] Text
moenning_final_submission.pdf - Examined Thesis (PDF)
Restricted until 6 September 2022.

Request a copy

Abstract

This thesis presents novel work on complex-valued neural networks applied to Natural Language Processing. We experimentally show the validity of complex-valued neural networks for semantic and phonetic processing of natural languages. We highlight important issues that complex networks have in comparison to their real-valued counter parts. In particular this work considers the tasks of Language Modelling, Semantic Similarity Judgement, Basic Question Answering, Phonetic Transcription and Automatic Speech Recognition. Our contributions are the translation of neural network building blocks to the complex plane and their experimental application in a variety of natural language tasks. We present criteria to compare real-valued and complex-valued neural networks for classification tasks. We present various complex embedding methods for words. These produce position and frequency-based word representations trainable using language models and usable in down-stream tasks. We also compare a real-valued and complex-valued memory network used for Question Answering. We derive a quantum-inspired framework for languages. Additionally, we demonstrate quantum-inspired Semantic Spaces. A general framework of complex-valued attention is presented in this thesis. It is used to derive spectral self-attention with a novel activation function. We also introduce two pooling functions to reduce dimensionality of frequency-based representations. A Spectral Transformer architecture facilitates the spectral self-attention for Speech Recognition. This work also includes a novel dataset for transcription of children's utterances consisting of seven sub tasks each with fixed data splits and baselines for better comparison and reproducibility. Throughout this thesis we find that complex-valued neural networks are suitable for natural language tasks, but require additional care in their design and training.

Item Type: Thesis (PhD)
Academic Units: The University of York > Computer Science (York)
Depositing User: Mr Nils Mönning
Date Deposited: 28 Oct 2019 12:49
Last Modified: 28 Oct 2019 12:49
URI: http://etheses.whiterose.ac.uk/id/eprint/24802

Please use the 'Request a copy' link(s) above to request this thesis. This will be sent directly to someone who may authorise access.
You can contact us about this thesis. If you need to make a general enquiry, please see the Contact us page.

Actions (repository staff only: login required)