Causal Surrogate Models: Adding "What If" to Cyber-Physical System Testing

Cyber-physical systems (CPS) are becoming more prevalent, especially in human-interacting environments. Identifying incorrect behaviour through software testing is, therefore, paramount to their use. Surrogate-assisted testing approaches aim to effectively test such systems by searching for scenarios that may result in system violations by using surrogate models to search for potential violations and evaluating those scenarios on a high-fidelity simulator. However, the development of surrogate models requires curated datasets to accurately represent system behaviour. Such datasets are typically unavailable for CPSs due to the limitations and expense of physical execution, especially for human-interacting systems. Pre-existing datasets may be used instead, which may contain spurious associations or may not cover all behaviours (a lack of controllability). Unmeasurable environmental factors may also affect system behaviour, making system outputs appear inconsistent for a given input (a lack of observability). In this thesis, we use a motivating example of an artificial pancreas system (APS) to investigate the limitations of testing such a system.

To account for the lack of observability and controllability of CPSs, we define a causal surrogate model to enable more effective testing of their behaviour. This surrogate model integrates with existing surrogate-assisted testing techniques. Our surrogate model uses causal inference, which can account for bias in pre-existing datasets and assess the expected causal relationships between variables. As a result, we enable the testing of systems for which curated data may not be available or that exhibit behaviours affected by external factors.

We perform two evaluations, first replicating an existing study of surrogate-assisted CPS testing on an automated driving system (ADS). In this evaluation, our results demonstrate how our approach found system violations with less computational expense than the state-of-the-art. We then test a more complex, safety-critical APS. However, to test the APS, we first develop and validate a digital twin of a person using an APS to act as a high-fidelity simulator. The APS can, therefore, be disconnected from the human-in-the-loop, allowing for the testing of potentially dangerous scenarios without clinical trials. Our causal surrogate model demonstrates the ability to uncover over double the number of violations using real-world clinical APS data, compared to the state-of-the-art surrogate-assisted testing approach. We show how causal surrogate models can alleviate the requirement of curated data for systems testing and present a novel way of navigating and finding system violations for inconsistent system behaviour