D5.1 – 3D modelling of anatomy and 3D registration

This deliverable will report on the set of tools for 3D modelling of the human anatomy and 3D registration from intra-operative (i.e. endoscopic images) and pre-operative data (e.g. MRI scans). The 3D reconstruction of the anatomy will be carried out by exploiting the most advanced 3D vision techniques, including stereo and multi-view geometry, Simultaneous Location and Mapping (SLAM) and non-rigid structure-from-motion. In this document we will present all the work that has been performed during the SARAS project to allow the system to automatically generate a 3D map of the anatomical environment. With respect to standard 3D reconstruction applications, our scenario presents specific challenges that make the problem extreme. First of all the structures (i.e. organs) we are interested in reconstruct are highly deformable, with no reliable dynamical models available in advance. Furthermore, the anatomical structures are connected to each other by means of fat, vessels and connective tissues with high variability in mechanical properties, which makes the behaviour of these connections unpredictable. Lastly, the variability of shape, dimensions and appearance of these structures is extremely high, so to make impossible to use a prototype model to be used for the majority of the subjects as prior knowledge. In the rest of this document we will provide details and discussions about all the main phases involved in this task, including system calibration, multiple view geometrical reconstruction and integration with the robot kinematics. In Section 2 we present all the technical aspects related to the calibration of the vision system, both in terms of camera calibration and camera-robot calibration to make the results of 3D reconstruction of the anatomical environment available in an appropriate reference frame, i.e. registered with the robotic arms operating in that workspace. The 3D reconstruction of the anatomy, including point cloud triangulation and structure from motion is presented in Section 3; monocular SLAM and 3D registration with respect to the a priori 3D map is detailed in Section 4. With respect to the original description of this deliverable available in the Grant Agreement, we decided to drop the automatic map update section. After receiving constructive feedbacks from medical doctors: a dynamical registration of a pre-operative map would become useless after only a few minutes due to the ever-changing nature of the anatomical scenario. We comment on this point in Section 5. Following the SARAS vision, in this work we only used available technologies (i.e. instruments) already certified and available as medical devices. Despite that, in Section 6 we discuss new technologies that can be useful in this task and their potential implementation into our project