Safety Assurance for Patient-Facing Clinical Conversational Agents (CAs)

Conversational Agents (CAs) are increasingly being utilised in healthcare to alleviate the burden on clinical resources. However, the integration of machine learning (ML)-based CAs into clinical settings introduces significant safety risks, as the non-deterministic behaviour of ML algorithms can create unpredictable failure modes that may compromise patient safety. Despite emerging literature on ML in healthcare, systematic safety assurance approaches specifically tailored for CAs remain underdeveloped. This thesis presents a safety assurance methodology for clinical CAs. First, a structured failure mode taxonomy for CAs is developed, identifying both technical and socio-technical causes. This taxonomy is then used to support a structured safety assurance framework. This framework demonstrates how to systematically conduct hazard analysis, derive a multi-layered set of safety requirements (for the system, the ML model, and its data), and construct a safety case using Goal Structuring Notation (GSN). The methodology is applied and evaluated through a series of progressively mature use cases, leading to a detailed case study of a real-world CA deployed in NHS hospitals. The research combines quantitative analysis of ML model performance with qualitative evaluation of the overall framework through stakeholder interviews. The findings suggest that the methodology helps identify context-dependent failures and improves traceability. Overall, the thesis contributes a structured methodology, evaluated through empirical case studies, for identifying hazards, analysing and managing risks, and implementing mitigations, thereby helping to bridge the gap between safety engineering and the practical deployment of ML-based clinical CAs.