ADVANCED STRATEGIES FOR AUGMENTED REALITY SIMULATION-BASED SURGICAL TRAINING

The augmented reality (AR) simulation represents a great potential for learning and transferring new skills, not only in minimally invasive surgery (MIS) but also in open surgery. AR simulators retain the natural haptic feedback of physical simulators and the performance evaluation tools of VR simulators. However, commercial AR simulators (e.g. CAE Healthcare ProMIS) are often based on physical models that are neither highly detailed, nor anthropomorphic and they have costs comparable to those of VR simulators. In addition, available commercial AR systems only provide visual instructions and guidance information to the trainee and do not allow either to update the virtual anatomy following deformations impressed on physical anatomical models, or to add simulated physiological functionalities which increase the realism of the system. This thesis proposes an advanced strategy to develop AR simulation platforms which combine highly detailed physical models and virtual reality information in a surgical scene, and integrates deeply both visual and tactile AR, and acoustic functionalities, improving the state of the art on AR simulators. The strategy can be applied to simulate all surgical procedures involving the task of identification and isolation of generic tubular structures, both in laparoscopic and open approach. The cholecystectomy, one of the most common surgical procedure, has been chosen as benchmark procedure to demonstrate the potentialities of the proposed strategy. Specifically, the simulator is designed for the task of identification and isolation of the Calot's triangle, the most critical phase of cholecystectomy. The proposed simulator includes all the anatomical structures which could be either seen or touched during the execution of the procedure: patient-specific physical replicas of liver, gallbladder, pancreas, abdominal aorta, esophagus-stomach-duodenum and realistic physical models of biliary ducts (BT), arterial tree (AT) and connective tissue. A key feature of AT and BT is the ability to be easily and separately replaced on demand, with different specific anatomical variations, thus realizing training sessions with increasing complexity. All the anatomical structures, except from the connective tissue which has to be dissected, are fabricated using materials which are extremely durable over time and reusable, such as silicone and nitinol tubes, thus reducing training costs. The BT and the AT are sensorized with Electromagnetic (EM) sensors and tracked in real time for the implementation of AR functionalities: tactile AR in open surgery and visual AR both in open and in laparoscopic surgery. The laparoscope is localized in real time, by means of an additional camera for tracking of a structured marker, in order to allow its maneuverability as commonly occur in laparoscopic procedures. The open surgery mode includes tactile AR functionality and a wearable haptic device to provide pulsed feedback during palpation, the latter tracked by means of an EM localization. AR is used as an aid to accurately show the Calot's triangle position, by means of AR visualization mode, in the laparoscopic approach, and to provide pulsed feedback during palpation, by means of AR tactile, in the open approach. This is possible thanks to the versatility of the proposed AR simulator which allows the transition from the laparoscopic to the open approach by replacing the laparoscope with a wearable haptic device. In addition, the simulator includes an electrical apparatus to report surgical errors. An attractive feature of this system is that it allows the user to interact in real-time with virtual models and to perceive the implemented organ’s functionality, blood vessels pulsation, in a natural way. Preliminary tests show that the AR simulator satisfies all the basic specifications: good anatomic appearance, modularity, reusability, cost-effectiveness, robustness, ability to report surgical errors and ability to provide AR aids. General surgeons positively evaluated the realism both the connective tissue and the AT and the BT embedded into the connective tissue. A positive qualitative feedback has been received regarding the usefulness of both acoustic functionality, to signal surgical errors, and of the AR scene, as an aid to detection of AT and BT. In conclusion, based on the studies performed in the context of this thesis and the users’ feedbacks, the AR simulator is likely to be considered as a potential training tool to learn the task of identification and isolation of anatomical tubular structures not clearly visible. In conclusion, as demonstrated in this thesis, a great step forward in surgical simulation will be possible developing hybrid AR simulators with a deep integration between real and virtual components