I Have No Mouth, and I Must Glean: Assessing Phonetically and Phonologically Motivated Language Understanding

Whilst language models have long been capable of generating grammatical and coherent text, their understanding of phonetically and phonologically motivated language remains weak. Primarily, this is due to such models being trained exclusively on orthographic text, losing much of the information contained in speech. This thesis proposes approaches to assessing and alleviating this phonetic and phonological deficit, regarding both explicit linguistic knowledge and the impact on downstream tasks such as creative language generation, humour explanation, and in silico sound symbolism experimentation. First, the thesis establishes a standard for the human evaluation of subjective language. By exploring the impact of demographic variables on the interpretation of such language, the thesis establishes best practices for assessing creative generation, such as the phonological wordplay that is explored in subsequent chapters. It then introduces the task of tongue twister generation to assess the ability of pre-trained language models to implicitly learn phonological patterns from orthographic text, introducing a phonologically informed generation pipeline, two novel tongue twister datasets, a constrained decoding module, and phoneme-level evaluation metrics. Following this, the thesis transitions towards assessing the understanding and reasoning of large language models, exploring their ability to explain the humour underlying different types of jokes, from phonologically inspired heterographic puns to long-form topical humour. Multimodal phonetic and phonological understanding is then assessed by replicating a range of classic psycholinguistic experiments regarding sound symbolism using vision language models, demonstrating their capacity for in silico experimentation. Multimodality is further explored through the novel task of spectrogram and waveform interpretation, assessing the integration of visual information with parametric linguistic knowledge. The thesis concludes with a comparative investigation into the perspectives of artificial intelligence practitioners and the general public, retrospectively assessing the desire for creative language generation and in silico experimentation, whilst exploring the wider societal impact of such technologies overall.