REAL-TIME DYNAMIC OBJECT GRASPING WITH A ROBOTIC ARM: A DESIGN FOR VISUALLY IMPAIRED PERSONS

Robotic arms have been increasingly used in different application areas such as manufacturing, automobility, and the medical industry. Research is also ongoing into how to utilize robotic arms to improve the lives of ordinary citizens who suffer from various forms of disabilities. For instance, it is often difficult for physically impaired individuals to complete tasks such as obtaining food or drink from the refrigerator and picking a needed item from a drawer. In such cases, a robotic arm could potentially be used to assist the visually impaired individual to pick up items. This work builds on existing research that utilizes audio commands, natural language processing, and object detection to determine the object of interest and potential grasp point in an image. This thesis builds on that work in multiple dimensions. First, the system is extended to work with streaming video from an Intel RealSense D455 camera and we propose and implement a technique to dynamically adjust the image data sent to the grasp detection module. Thirdly, we translate the grasp point coordinates from the camera coordinate system to the robotic arm coordinate system and finally integrate the solution with a Dobot Magician robotic arm. We successfully tested the solution and showed that given the relevant command e.g. pick a blue bottle, using the camera and robotic arm our solution can identify the blue bottle, a grasp point, and pick the bottle when it is placed randomly in a given area