Multi-object Localization in Robotic Hand

We have developed a machine learning approach to localized objects inside a robotic hand using only images from 2D cameras. Specifically, we used deep learning method (You Only Look Once, YOLO) and Iterative closest Point (ICP) to estimate the 3D coordinates of the objects in a robotic hand. This method will also output the number of objects inside the robotic hand in addition to the coordinates of the objects. We have demonstrated the performance with simulation and obtained typical accuracy within a few pixels (couple mm) and counting accuracy of about 76%. We have also applied it to real images, which is currently a work in progress to improve prediction performance. Furthermore, we are in the process of expanding the model to predict objects other than spheres. Our approach can find applications in many image-based object localization applications including industrial and service robotics