Data-driven Spatially and Temporally Consistent Scene Representations for Robotic Minimally Invasive Procedures

Robotic Minimally Invasive Procedures (MIP) generate large amounts of intraoperative data, including endoscopic video and instrument motion. Despite technological advances, current systems lack the ability to build a consistent and structured understanding of the surgical scene. This thesis addresses this limitation by investigating how visual information in robotic MIP can be organised into scene representations that remain coherent across space and time. The central premise is that surgical understanding depends not only on isolated task performance, but on how different visual representations (from object localisation to three-dimensional reconstruction) interact and support one another.

The thesis first introduces a representation-centric framework that formalises visual surgical perception as a multi-level process, clarifying how detection, tracking, motion estimation, and reconstruction are connected. A transformer-based detection model is then proposed to obtain robust object representations under realistic perturbations. Moving on, render-based implicit scene representations are developed to jointly model appearance, motion, and three-dimensional structure in deformable environments, enabling temporally consistent tracking and separation of camera motion from tissue deformation. Finally, the DRENDS dataset is presented to support quantitative evaluation of temporally consistent depth estimation in dynamic surgical scenarios.

Overall, this work demonstrates that effective surgical perception depends on how visual information is structured and integrated, and it contributes foundational methods and evaluation tools for Surgical Data Science.