Dog's Dinners and Copy Cats: Human and Language Model Processing of Idiomatic Expressions

Idioms are multi-word constructions with unusual semantics; their meaning differs from those of their component words. A big cheese, for example, need not be physically large and is unlikely to literally be cheese!
While such idiomatic expressions are widespread as a class and are found throughout the world's languages, individual idioms are comparatively rare. These properties make them challenging (and interesting) for humans learning a language, translating between languages or developing linguistic theories.

Computational language models also struggle to adequately represent and process figurative language such as idioms, displaying a marked tendency to blend elements of the literal and idiomatic interpretations and lack of responsiveness to disambiguating context.

This thesis explores processing of idioms by humans and language models, comparing and contrasting them when exposed to stimuli representing potentially idiomatic expressions in text and images. It presents evidence that human readers briefly revisit earlier words more frequently in idiomatic instances than literal ones, and that the effects of semantic incongruity are similarly asymmetrical.
Pre-trained language models are shown to be less responsive to stimuli which mix literal and figurative elements than humans, and their processing of idioms to be less similar to human reading than general language. These effects persist despite increases in the size of models and training datasets and in their general language capabilities.

In a (newly introduced) multimodal setting, factors such as semantic compositionality and concreteness are shown to be more strongly related to performance for vision-language models than for humans. Literal instances are simpler for humans and especially for models, which are prone to blending elements of literal and idiomatic interpretations.

While the primary focus is on English, idioms in Brazilian Portuguese are also studied.