Creating a Digital Twin of Spinal Surgery:
A Proof of Concept

Jonas Hein1,2, Frédéric Giraud1, Lilian Calvet1, Alexander Schwarz2, Nicola Alessandro Cavalcanti1, Sergey Prokudin2, Mazda Farshad1, Siyu Tang2, Marc Pollefeys2, Fabio Carrillo1, Philipp Fürnstahl1
1Balgrist University Hospital, University of Zurich, Switzerland,
2ETH Zurich, Switzerland,
arXiv Paper Poster Video VR Demo Video VR Demo Report
Digital photograph of a spinal surgery Rendering of its digital twin

A proof-of-concept for surgical digital twins.

Abstract

Surgery digitalization is the process of creating a virtual replica of real-world surgery, also referred to as a surgical digital twin (SDT). It has significant applications in various fields such as education and training, surgical planning, and automation of surgical tasks. In addition, SDTs are an ideal foundation for machine learning methods, enabling the automatic generation of training data. In this paper, we present a proof of concept (PoC) for surgery digitalization that is applied to an ex-vivo spinal surgery. The proposed digitalization focuses on the acquisition and modelling of the geometry and appearance of the entire surgical scene. We employ five RGB-D cameras for dynamic 3D reconstruction of the surgeon, a high-end camera for 3D reconstruction of the anatomy, an infrared stereo camera for surgical instrument tracking, and a laser scanner for 3D reconstruction of the operating room and data fusion. We justify the proposed methodology, discuss the challenges faced and further extensions of our prototype. While our PoC partially relies on manual data curation, its high quality and great potential motivate the development of automated methods for the creation of SDTs.

Video

VR Demo Application

Our VR application of this project was accepted as a demo at ISMAR 2024!

Assets

The assets will be available here soon.

BibTeX

@InProceedings{Hein_2024_CVPRW,
  author    = {Hein, Jonas and Giraud, Fr\'ed\'eric and Calvet, Lilian and Schwarz, Alexander and Cavalcanti, Nicola Alessandro and Prokudin, Sergey and Farshad, Mazda and Tang, Siyu and Pollefeys, Marc and Carrillo, Fabio and F\"urnstahl, Philipp},
  title     = {Creating a Digital Twin of Spinal Surgery: A Proof of Concept},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  month     = {June},
  year      = {2024},
  pages     = {2355-2364},
  keywords={Surgery Digitization;Surgical Digital Twin;Digital Twin;Surgical Data Science;Surgery;Digitization;3D Reconstruction;Photogrammetry;3D Scanning;Laser Scan;Motion Capture;Pose Estimation;Human Pose Estimation;Surgical Instruments;Instrument Tracking;Pose Estimation;OR-X},
}
@InProceedings{Hein_2024_ISMAR,
  author    = {Hein, Jonas and Grunder, Jan and Calvet, Lilian and Giraud, Fr\'ed\'eric and Cavalcanti, Nicola Alessandro and Carrillo, Fabio and F\"urnstahl, Philipp},
  booktitle = {2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)},
  title     = {Virtual Reality for Immersive Education in Orthopedic Surgery Digital Twins},
  year      = {2024},
  volume    = {},
  number    = {},
  pages     = {},
  keywords  = {Virtual Reality;Surgery Digital Twin;Surgical Training;Medical Simulation;Spinal Surgery},
}

Acknowledgements

This work has been supported by the OR-X - a swiss national research infrastructure for translational surgery - and associated funding by the University of Zurich and University Hospital Balgrist, as well as by the InnoSuisse PROFICIENCY grant.