A 4D ultrasound imaging automation platform for modelling and assessment of ultrasound target dynamics using direct visual servoing and machine learning

Purpose: The main objective of this research project is to address the limitations of image-processing computational performance of current ultrasound target motion estimation techniques by developing a novel high-performance ultrasound motion estimation technique that eliminates the need for using currently adopted image-registration-based motion-estimation techniques for systems requiring ultrasound target tracking. To address the system's dependence on the operator's experience and supervision as the second objective of this research project, the developed target tracking technique is integrated with a novel automation platform developed to support medical physicists with tools for collectively implementing, supervising, and training arbitrary ultrasound target motion estimation tasks and incorporate them in the platform as reusable solutions to support qualified and reproducible research. Methodologies: The main objective of this research project has been addressed by enabling high-performance and accurate motion estimation of soft-tissue-equivalent ultrasound targets using the ultrasound direct visual servoing (US-DVS) technique for the first time. The inaccurate US-target tracking capabilities of the implemented US-DVS technique have been addressed and optimised using specially trained machine-learning models which lead to the US direct visual modelling (US-DVM) technique introduced by this project. The machine-learning models implemented by the US-DVM technique were trained using two types of US-target motion simulations: simulation of tissue-equivalent US-target motion based on predefined motion trajectories provided by high-precision robotic-arms; and simulations using an ultrasound digital-target dynamics simulator (US-DTDS) developed by this project. The second objective of this research project has been addressed by maximising the performance and target detection accuracy of the proposed motion estimation technique by minimising its dependence on the operator and by transferring the operator's target scanning and identification experience to a provenance-enabled automation system. To Achieve this, another machine-learning model, based on Gaussian mixture modelling (GMM), was also developed and used to improve the performance of standard image segmentation techniques allowing for automating target detection and tracking in real-time. In addition, the interaction of the operator with the platform has been optimised by employing a provenance-enabled workflow automation framework, called VisTrails, to implement the proposed new technique and all the supporting services required. This will help in learning the operator's skills by the automation workflow, which will minimise the operators' systematic errors and support reproducible research. Results: A Medical Physics Services Framework (MPS-F) has been developed based on VisTrails and used to control two robotic arms to track and capture the simulated motion using the US-DVM technique. The new US-DVM technique has enabled accurate estimation of ultrasound target motion from US-DVS tracking feedback with accuracies better than 1.5%, and with computational performance up to 14 times better than motion estimation techniques used in current practice. This allows for more accurate real-time tracking of ultrasound targets like the prostate. Regarding the second objective, high-performance detection and extraction of deformable ultrasound targets using the morphological active-contour technique (MACT) has been achieved for average prostate-size targets to within 0.04 seconds using down-sampling strategies. To further support the second objective, the MPS-F has been developed as a provenance-enabled software automation solution to enable other researchers to reuse and reproduce most of the implementations and results of this research, and hence minimise the operator's systematic errors. The MPS-F has been used to implement a novel technique that combined the MACT and the US-DVM to detect and estimate prostate-size target deformations with volume accuracies better than 0.005 cm^3. Conclusion: This research study introduced the US-DVM technique as a novel ultrasound target motion modelling and estimation solution based on ultrasound direct visual servoing (US-DVS). US-DVS is used typically for ultrasound target tracking in real-time and using it for modelling and estimating target dynamics has been addressed for the first time by this study based on machine learning strategies. The machine learning approach provided solutions that overcame inherent limitations of the imaging system allowing for predicting faster dynamics and larger interframe displacements, in addition to enabling supervised and automated predictions of accurate motion estimations tailored specifically for individual targets for optimal consistency and accuracy. The proposed techniques and solutions have been implemented and evaluated by the medical physics services framework (MPS-F), which was developed by this project based on the VisTrails workflow management system. The MPS-F is considered a major contribution by this study where it enables the operator to implement, train, and supervise arbitrary medical physics workflows that can employ machine-learning, image-processing, and ultrasound target tracking and motion estimation tasks. With the help of the MPF-S, it is proposed by this study that an ultrasound modelling and automation platform (US-MAP) can be constructed as a more efficient real-time alternative for the Elekta Clarity Autoscan system being the only non-telerobotic real-time ultrasound target tracking system implemented clinically for tracking prostate motion