Investigating the Relationships Between Mutants, Oracles and Pseudo-Testedness

Test suites are critical in developing high-quality software, yet are commonly measured
using code coverage. Code coverage measures the quantity of test suite-executed code,
without requiring code behaviours be checked by an oracle (e.g., an assertion), leaving
code vulnerable to pseudo-testedness. Pseudo-tested code is executed by the test suite,
yet can be deleted without triggering test failures. Intuitively, pseudo-testedness can
be detected using deletion operators, but has not been explored at the statement level.
The relationship between pseudo-tested statement identification (PTSI) and other oracle-
based adequacy metrics, i.e., checked coverage, remains unclear, leaving open questions
about each technique’s role. Because PTSI relies on rule-based deletion mutation, it
only highlights the absence of a check. To move beyond identifying where checks are
missing, mutants must produce semantically incorrect behaviours that reveal the context
requiring the check. While large language models (LLMs) enable context-aware mutation
testing, current LLM-based tools are constrained to small code changes, such as lines
or AST nodes. To utilise the LLM’s capabilities, tools must move towards algorithm-
level approaches that guide the LLM to produce non-trivial, productive mutants. This
thesis aims to characterise these relationships and utilise them to refine the techniques.
First, this thesis explores pseudo-testedness across 23 projects, identifying that pseudo-
tested methods miss 48% of pseudo-tested statements. Secondly, compared with checked
coverage using dynamic and observation-based slicers, PTSI locates code with the lowest
median mutation scores (0.32 vs. 0.76 and 0.5) while executing at least 10x faster. Finally,
this thesis uses systematic hazard analysis to guide LLMs in generating algorithm-level
mutants that are uniquely subsuming and defect-matching compared with existing LLM-
and rule-based techniques. By characterising the relationships between mutants, oracles
and pseudo-testedness, this thesis enables the refinement of these techniques for test suite assessment.