Towards Automated Suturing of Soft Tissue: Automating Suturing Hand-off Task for da Vinci Research Kit Arm using Reinforcement Learning

Successful applications of Reinforcement Learning (RL) in the robotics field has proliferated after DeepMind and OpenAI showed the ability of RL techniques to develop intelligent robotic systems that could learn to perform complex tasks. Ever since the use of robots for surgical procedures, researchers have been trying to bring some sort of autonomy into the operating room. Surgical robotic systems such as da Vinci currently provide the surgeons with direct control. To relieve the stress and the burden on the surgeon using the da Vinci robot, semi-automating or automating surgical tasks such as suturing can be beneficial. This work presents a RL-based approach to automate the needle hand-off task. It puts forward two approaches based on the type of environment, a discrete and continuous space approach. For capturing a unique suturing style, user data was collected using the da Vinci Research Kit to generate a sparse reward function. It was used to derive an optimal policy using Q-learning for a discretized environment. Further, a RL framework for da Vinci Research Kit was developed using a real-time dynamics simulator - Asynchronous Multi-Body Framework (AMBF). A model was trained and evaluated to reach the desired goal using model-free RL techniques while considering the dynamics of the robot to help mitigate the difficulty in transferring trained model to real-world robots. Therefore, the developed RL framework would enable the RL community to train surgical robots using state of the art RL techniques and transfer it to real-world robots with minimal effort. Based on the results obtained, the viability of applying RL techniques to develop a supervised level of autonomy for performing surgical tasks is discussed. To summarize, this work mainly focuses on using RL to automate the suture hand-off task in order to move a step towards solving the greater problem of automating suturing