Learning the semantics of object-action relations by observation

Recognizing manipulations performed by a human and the transfer and execution of this by a robot is a difficult problem. We address this in the cur-rent study by introducing a novel representation of the relations between objects at decisive time points during a manipulation. Thereby, we encode the es-sential changes in a visual scenery in a condensed way such that a robot can recognize and learn a manipula-tion without prior object knowledge. To achieve this we continuously track image segments in the video and construct a dynamic graph sequence. Topologi-cal transitions of those graphs occur whenever a spa-tial relation between some segments has changed in a discontinuous way and these moments are stored in a transition matrix called the semantic event chain (SEC). We demonstrate that these time points are highly descriptive for distinguishing different manip-ulations. Employing simple sub-string search algo-rithms, semantic event chains can be compared and type-similar manipulations can be recognized with high confidence. As the approach is generic, statisti-cal learning can be used to find the archetypal SEC of a given manipulation class. The performance of the algorithm is demonstrated on a set of real videos showing hands manipulating various objects and per-forming different actions. In experiments with a robotic arm, we show that the SEC can be learned by observing human manipulations, transferred to a new scenario, and then reproduced by the machine