[2201.00383] Integrating Artificial Intelligence and Augmented Reality in Robotic Surgery: An Initial dVRK Study Using a Surgical Education Scenario

Robot-assisted surgery has become progressively more and more popular due to
its clinical advantages. In the meanwhile, the artificial intelligence and
augmented reality in robotic surgery are developing rapidly and receive lots of
attention. However, current methods have not discussed the coherent integration
of AI and AR in robotic surgery. In this paper, we develop a novel system by
seamlessly merging artificial intelligence module and augmented reality
visualization to automatically generate the surgical guidance for robotic
surgery education. Specifically, we first leverage reinforcement leaning to
learn from expert demonstration and then generate 3D guidance trajectory,
providing prior context information of the surgical procedure. Along with other
information such as text hint, the 3D trajectory is then overlaid in the stereo
view of dVRK, where the user can perceive the 3D guidance and learn the
procedure. The proposed system is evaluated through a preliminary experiment on
surgical education task peg-transfer, which proves its feasibility and
potential as the next generation of robot-assisted surgery education solution.