Deep Latent Variable Models for Text Modelling

Deep latent variable models is a class of models that parameterise components of probabilistic latent variable models with neural networks. This class of models can capture useful high-level representations of information from the input data, and has been widely applied to many domains (e.g., images, speech, and texts), with tasks ranging from image synthesis to dialogue response generation.

For instance, implicit linguistic cues such as topic information are helpful for various text modelling tasks, e.g., language modelling, dialogue response generation. Being able to accurately recognising dialogue acts plays a key role to help generate relevant and meaningful responses for dialogue systems. However, existing deep learning models mostly focus on modelling the interactions between utterances during a conversation (i.e., contextual information), where important implicit linguistic cues (e.g., topic information of the utterances) for recognising dialogue acts have not been considered. This motivates our first model, which is a dual-attention hierarchical recurrent neural network model for dialogue act classification. Compared to other works which focus on modelling contextual information, our model considers, for the first time, both topic information and dialogue act using a dual-attention hierarchical deep learning framework. Experimental results show that our model achieves a better or comparable performance than other baselines.

When applying deep latent variable models in the text domain, one can generate diverse texts via randomly sampling latent codes from the trained latent space. However, several noticeable issues of deep latent variable models in the text domain remained unsolved, where one of such issues is KL loss vanishing and has serious effects on the quality of generated texts. To tackle this challenge, we propose a simple and robust Variational Autoencoder (VAE) model to alleviate the KL loss vanishing issue. Specifically, a timestep-wise KL regularisation is proposed and imposed into the encoder of VAE at each timestep. This method does not require careful engineering the objective function of VAE or constructing a more complicated model architecture, as existing models do. In addition, our approach can be easily applied to any types of RNN-based VAEs. Our model is evaluated in the language modelling task and successfully alleviates the KL loss vanishing issue. Our model has also been tested on the dialogue response generation task, which not only avoids the KL loss vanishing issue, but also generates relevant, diverse and contentful responses.

Finally, we investigate the low-density latent regions (holes) of VAE in the text domain, a phenomenon which exists in the trained latent space of VAE and leads to low-quality outputs when latent variables are sampled from those areas. In order to provide an in-depth analysis of the holes issue, a novel and efficient tree-based decoder-centric algorithm for the low- density latent regions identification is developed. We further explore how the holes impact the performance of generated texts of VAE models. For instance, we analyse whether the holes are really vacant, which captures no useful information and how the holes are distributed in the latent space.