Multi-modal vision techniques for image-guided surgery

During minimally-invasive surgery, endoscopes and other surgical tools enter the body of the patient through small openings, allowing the operations to be performed with less post-operative pain and faster recovery for the patient, as well as less wound complications. Although great advantages are achieved from the patient’s point-of-view, several difficulties have still to be handled by the surgeons. First, the limited field of view of the endoscope makes target-area localizations and related identifications complex. Second, surgeons should still match mentally the medical images for the surgical planning with the current patient anatomy. These difficulties can be addressed by using computer vision technologies for guiding surgical procedures. More specifically, this thesis aims at the following three points, the first two for neurosurgery, and the third for perfusion assessment during surgery. The challenges for this dissertation are partitioned in two primary aspects. The first point aims at the fusion of medical images on the endoscopic view, in order to develop an augmented reality system. The second point is based on hyperspectral imaging (HSI), which is compared with diffuse reflectance spectroscopy (DRS), to improve brain tissue classification for neurosurgery. The third point exploits the PPG imaging (iPPG) technique and its potential use is evaluated for peripheral arterial disease (PAD) and organ perfusion assessment during surgery. To address the first point, a new neurosurgical application is implemented on the already existing Philips Augmented Reality (AR) surgical navigation system, designed for spinal surgery. This navigation system incorporates an optical tracking system (OTS) with four video cameras embedded in the flat detector of the motorized C-arm. A hand-eye camera calibration algorithm for the fusion of medical images on the endoscopic view is implemented and integrated into the AR system. This technology is validated for endo-nasal surgery, a neurosurgical procedure for the removal of tumors located at the skull base, such as pituitary tumors. Intra-operative Cone Beam Computed Tomography (CBCT) images are fused with the view of the surgical field obtained by the endoscope camera. The accuracy of CBCT image co-registration is tested, using a custom-made grid with incorporated 3D spheres. The system achieves a sub-millimeter accuracy of image overlay of 0.55 mm, measured as mean target registration error (TRE), with a standard deviation of 0.24 mm. Afterwards, an anatomically realistic head phantom is developed, with materials chosen to achieve both X-ray attenuation and mechanical properties, similar to the real tissue. Using the phantom, a proof of concept of the skull-base surgical simulation is provided, then the accuracy and efficacy of the AR system are evaluated for the insertion of biopsy needles, which is a common neurosurgical procedure that requires high precision. Several 2-mm spherical biopsy targets are inserted inside the brain of the brain phantom. The obtained mean accuracy of the biopsy needle insertions (n=30) is 0.8±0.43 mm, with a mean device insertion time of 155±43 seconds. These experiments demonstrate that a high accuracy is obtained during neurosurgery with the proposed methods and one phantom. For the second point, a near-infrared (NIR) hyperspectral imaging (HSI) sensor is mounted on an endoscope, to explore contactless brain-tissue classification and HSI is compared with diffuse reflectance spectroscopy (DRS), which is an alternative optical technique that requires a probe in contact with the tissue. The classification is performed on ex-vivo porcine brain tissue, which is analyzed and classified in white and gray matter. The HSI reaches a sensitivity of 95% and specificity of 93%, whereas DRS reaches sensitivity and specificity of 96%. The results show that the spectral signature of the tissue in the NIR range contains sufficient information to discriminate brain tissue in white/ gray matter. Further investigation on ex-vivo tumor sample data is required prior to the clinical validation of HSI. The third part concentrates on exploring PPG imaging for extracting perfusion information on tissue. By using an off-the-shelf camera and a light source, the dynamic changes in blood volume are remotely detected beneath the skin and a map is derived correlated to the blood perfusion. After evaluating PPG imaging for local and temporal perfusion-change detections, it is employed for Peripheral Arterial Diseases (PAD) assessment. Reduced blood flow is simulated on 21 volunteers and iPPG is compared with ultrasound and Laser Speckle Contrast Analysis. These experiments show that iPPG can detect reduced perfusion levels and correlates well with the other measurement systems. Finally, this technology is deployed for organ perfusion assessment during intestine surgery. The experiments demonstrate that PPG imaging can be successfully used for extracting perfusion maps from the organ surface, even for detecting perturbations and perfusion changes during several stages of the surgery. The results of this dissertation contribute with novel techniques and approaches to add value to endoscopic and navigation technology systems, as well as to tissue classification and perfusion monitoring, during minimally-invasive surgery. The three main explored research points and their proposed techniques, namely image fusion with the endoscopic view, tissue classification via HSI and PPG imaging for perfusion assessment, can be potentially implemented and combined into a single endoscopic platform, resulting into a new multi-modal endoscopic system. Moreover, the thesis shows that the proposed techniques and algorithms increase the quality of the decision-making process and have the potential to improve the patient surgical outcome