Artificial intelligence for the detection of recurrence in cancer patients

The recurrence and progression of cancer represent critical junctures in a patient’s
care and valuable opportunities to improve future treatment. Unfortunately, these
clinical endpoints are poorly recorded in healthcare datasets, requiring manual abstraction from unstructured clinical notes. Prior research into automating their
detection has focused on common cancers, first-recurrence events, simplistic methods, and has lacked access to curated datasets for external validation. This thesis
addresses these gaps through a series of case studies using Electronic Health Record
(EHR) data from patients with Epithelial Ovarian Cancer (EOC). It demonstrates
that while rule-based methods that search for disease-free intervals can identify a
limited subset of worsening cancer events, they are insufficient to detect successive events beyond the first recurrence. By contrast, simple data-driven Machine
Learning (ML) approaches, including Classification and Regression Tree (CART),
Random Forests, and gradient-boosted trees, effectively capture the relationship
between routinely collected data and successive, clinically relevant recurrence and
progression events. These events were found to be detectable from local features,
with the inclusion of features encoded by recurrent neural networks, which capture long-term dependencies, providing no significant improvement over simple ML
approaches using localised information. Finally, externally validating UK-trained
models on a US institution highlighted the central challenge in this domain: the
need for a consensus on what defines a clinically meaningful endpoint. Although
future work may refine model performance by rewarding detections that occur close
to the true event or by using earlier detections to support the identification of later
events, these technical advances cannot resolve the core limitation: the absence of
a globally recognised definition of what characterises a clinically relevant worsening
of a patient’s cancer. This lack of consensus not only hinders the development and
deployment of automated detection models but also undermines current research
aimed at improving the treatment of cancer patients.