Semantic Matching in Human-Robot Collaboration

Human-robot interaction, collaboration, and joint planning have specific limitations, due to the constraints of human communications. The solution to the general problem of communication is distant, but there are tasks, such as navigation that are within reach. A necessary component is the semantic map, where occupied voxels of the environment have labels, such as “table”, “chair”, “staircase”, etc. Previously, we have developed (with colleagues) a real-time semantic mapping system. It was evaluated on small robots. A general issue appeared: the labels differed for the robot, since the deep learning labelling algorithm could not provide outputs due to the very different viewpoint. The same problem should occur for drones, too. This calls for semantic matching between the human and the robot, where human-defined labels should be transferred to the robot’s perspective. In addition to correct labelling, understanding the geometry of any environment is essential for a deeper comprehension of the human’s environment. We present a system that can perform such semantic matching efficiently, using segmentation from any viewpoint utilizing superpixels and the 3D reconstruction obtained from a human perspective