Development and Evaluation of a Visual Feature Recognition Framework for a Robotic Assembly System

Today miniaturization has become a major development factor in di_erent industries such as electronics, medical technology, and watch making. MIRACLE project of the University of Basel is a project focusing on developing robotic endoscope devices for minimally invasive surgery. In this prototypes of medical devices were developed with millimeter dimensions and even micrometers. The developed prototypes need to be assembled, and their sizes are in millimeters or even micrometers. These small objects can get damaged when untrained humans assemble micro-objects. The teleoperated robotic micro-assembly station also developed within the MIRACLE project is an assistance robot for helping users to assemble fragile micro-parts. This system is based on teleoperated, force-sensitive, miniature robot arms, which handle the micro-parts and are controlled by the user through haptic interfaces. The users need to have visual information to complete tasks with high performance. Knowing the orientation of object is necessary because when two objects such as screw and nuts are not aligned, they would be got jammed. This master thesis focuses on providing the orientation of objects in order to tackle this challenge. During the development process, the list of micro-parts found in typical micro-assembly applications for estimating their orientation was identifed. The object list was divided into the electronic applications, mechanical wrist watch, and medical devices. The object list contains many different types of objects and we expect we will face the new object which is not in the object list. Therefore, we decided to develop pose estimation methods with respect to recognizing the 6D pose (orientation and translation) of unknown and known objects. Also, we decided to choose a machine learning method for estimating object pose due to developing high accuracy architecture. We designed the machine learning architecture based on the human mind's pose recognition with regards to developing an architecture for estimating the unknown object pose. We know that how it works with humans. They never see objects, and they need to know their poses, they search for a similar object in their memory to substitute it as an unknown object. For example, they substitute a cylinder as a resistor object if they never see the resistor. Then, they substitute the orientation of the cylinder for resistance. Also, if the object does not look like a simple shape, they search for object holes. Because screws and pins are inserted in the holes for assembling. Therefore, they do not need to recognize objects, the hole pose is enough to know for assembling tasks. We designed machine learning with respect to human mind architecture. After object recognition and classification, their keypoints should be recognized in order to estimate their poses. When the designed network detects object keypoints, these keypoints could be connected with each other to obtain the object coordinate system. The keypoints contain origin of object, x axis direction, y-axis direction and z-direction of objects. To calculate 6D pose estimation, knowing the camera coordination is necessary. With regards to camera calibration, the camera coordination was estimated. The translation and rotation were computed with knowing the camera and object coordinations. The method was evaluated in three steps. First, the error of camera calibration was computed and it was successful with reprojection errors of about 8 %. After that, the result of object detection for known and unknown objects was presented, with relatively high accuracy. In the last section, the result of keypoints detection was assessed. It was feasible but not robust enough yet . It was shown that the detecting object coordinate depends on keypoints detection, and therefore the 6D pose object was estimated regard to object coordinate. We recommend training network with more datasets to improve the robustness of the network. Besides, it would be better to train other objects like cube and gears to cover more objects in the object list. Finally, it would be a good idea to use other DNN network such as 3D reconstruction, or corner detection for detecting keypoints with better precision