Energy-based models for speech synthesis

Speech synthesis is an essential technology in a range of widely used applications. Recent progress in speech synthesis has largely been driven by autoregressive (AR) and non-autoregressive (NAR) models that generate highly natural speech but can still produce inconsistencies. This thesis is motivated by the need to address these inconsistencies using generative models, particularly energy-based models (EBMs), and investigates how EBMs can be made practical by connecting them to related frameworks such as diffusion and flow-matching methods. This thesis presents a study on applying energy-based models for speech synthesis. The primary objective of this research is to train energy-based TTS models with commonly used noise contrastive estimation (NCE). During this development the close connection between EBMs and diffusion models developed at the same time became apparent. Building on this connection, score matching, which does not rely on noisy samples, was investigated as an alternative training approach. However, score matching approaches, such as sliced score matching (SSM), is complicated to implement and requires careful tuning. A new loss function is proposed in this thesis to simplify SSM and achieve similar performance. This loss function has close connection to flow matching models which further strengthens the link between EBMs and diffusion like models. The results presented in this thesis show that the proposed methods can effectively improve the quality of synthesized speech compared to pre-trained TTS models.