Teaching Robot Companions to Assist Humans via Natural Language and Gestures

Workplaces in manufacturing contexts, especially automotive assembly, often require the transport of heavy parts, and today many of those lifts are done by pre-coded machines or people. However, in a more dynamic setting, it may be required for the robot to go off script, so it may stop on command in wait for another part to be added or removed for any adjustments needed. The pre-coded machines do not have this dynamic, and some parts are simply too heavy for people. To this end, we develop a teaching-learning framework for the robot to learn from multi-modal human demonstrations to assist its human partner in collaborative tasks. By taking advantage of our approach, humans can teach robots just like teachers teach students how to carry heavy parts via natural language and gestures. The Myo Armband is employed to acquire human gestures and parametrize human driving modes to train the robot. We then use natural language processing and structured dialogue to communicate with the robot combined with the wearable sensing to create a functioning, dynamically moving robot. Afterwards, the robot can learn what each driving mode is through the Random Forests (RF) algorithm. The proposed approach is implemented on a smart companion robot, which assists the human in carrying a heavy part around an assembly line. Testing results suggest that the human labor can be reduced by using natural language and wearable sensing, and the robot can effectively collaborate with the human to accomplish the shared task in collaborative manufacturing contexts