Understanding how Language Models Learn Under Constrained or Noisy Settings

Transformer-based Language models (LMs) have become the foundation of modern Natural Language Processing (NLP), demonstrating strong performance across various tasks. Typically trained on massive datasets with curated objectives and clean supervision, these models are increasingly being deployed in real-world settings where data is noisy, incomplete, or otherwise imperfect. Understanding what drives their learning and how robust they are to such conditions is essential.

This thesis investigates how different training signals, namely pre-training objectives, input structure, and instruction quality, affect the generalization behavior of LMs. Through three empirical studies, it examines model performance when traditional assumptions are relaxed or deliberately challenged. The first study explores the role of the pre-training objective, comparing linguistically intuitive objectives with arbitrary or non-linguistic alternatives. Surprisingly, even non-intuitive objectives yield models that acquire linguistic knowledge, suggesting that architecture and data distribution may matter more than the objective’s semantic alignment. The second study focuses on the internal structure of input tokens. Motivated by psycholinguistic findings, we test whether models can learn effectively from inputs with only partial character information. Results show that models remain robust, even with severely reduced inputs, indicating a surprising resilience to information loss. The final study shifts to instruction-tuning. Contrary to the common assumption that clean, well-formed prompts are necessary for generalization, we find that training with noisy or perturbed instructions can often improve both robustness and performance. This suggests that strategically applied noise can act as a form of regularization.

Together, these findings deepen our understanding of how LMs learn under constrained or imperfect conditions. They show that models can be not only effective when trained on clean, large-scale data but also surprisingly resilient, and at times improved, when faced with noisy or incomplete signals. This thesis offers practical insights for building robust, data-efficient language models suitable for real-world deployment.