Navigating Multimodal Complexity: Advances in Model Design, Dataset Creation, and Evaluation Techniques

Ibn Sina, a philosopher of 11th-century Persia, wrote of a `Floating Man'. This man is floating through a void, without the use of his sight or touch or any of the senses which make us human. Yet as he has a human brain this man, according to Ibn Sina, is capable of imagining and reasoning with the capabilities of any other person. With the development of Large Language Models the field of Artificial Intelligence has come close to making a `Floating Man' - or at least making a `Floating Man' with memories of more books than exist in the wildest dreams of the librarians of Alexandria or Oxford. In this thesis, we question if the `floating man' of AI could benefit from more of his senses, reasoning that as humans a great deal of our experience is multimodal. Our research aims to address the limitations of current NLP models that heavily rely on textual information, often at the expense of multimodal cues. Such errors highlight the critical need for multimodal approaches in many applications, of which we study Visual Question Answering, Citation Recommendation, and Eye-Tracking Prediction, where text alone can lead to biased, harmful, or simply incorrect outcomes, such as mistaking a metal table for one made of wood due to textual biases. Through our research, we aim to show the potential for Multimodality in enriching the capabilities of Artificial Intelligence.