Designing Games to Collect Human-Subject Data

Applied games align the 'fun' of gameplay with real-world outcomes to achieve social good. For data collection outcomes (e.g. where games are used for experiments or citizen science) various templates and taxonomies for design have been proposed to achieve this alignment. However, existing approaches assume it is always possible to evaluate (and validate) collected data against either 'ground truth' or intersubjective consensus. On the contrary, a significant proportion of human-subjects research is concerned with datums that cannot be validated in this way, such as latent traits and beliefs (e.g. ice cream preference, which cannot be 'validated' against a correct value). Despite extensive knowledge from the social science methodological literature, we do not have comparable templates or taxonomies that can help to design and analyse data collection games for these kinds of data: we cannot yet turn experiments into 'elicitation games'.

This thesis develops a theoretical model for such `elicitation games', using language elicitation as a case study. Elicitation games must satisfy requirements of validity and motivation. First, I survey validity threats characteristic of the use of games in experiments. Second, I construct a grounded theory of speech motivation to understand what motivates data-providing behaviours in applied games. Integrating these, I theoretically justify a generalised model of data elicitation in games: Intrinsic Elicitation. Finally, to identify which validity threats are of primary importance within this model, I run a series of controlled experiments comparing accuracy rates using a novel elicitation game for eliciting adjective order.

This thesis contributes a framework for integrating game design and social science experimental concerns and how they may influence each other for the design and analysis of elicitation games. I find that games incentive rational players to misalign data to experimental outcomes. This can be solved by novel game designs that follow Intrinsic Elicitation.