Learning Task- and Touch-based Grasping

In order to equip robots with goal-directed grasping ability, the integration of high-level task information with low-level\ua0sensory data is needed. For example, if a robot is given a task, e.g., pour me a cup of coffee, it needs to 1) make decision\ua0on which object to use, 2) how the hand should be placed around the object, and 3) how much gripping force should be\ua0applied so that the subsequent manipulation is feasible and stable for the pouring action. Several sensory streams (visual,\ua0proprioceptive and haptic) are relevant for these three steps. The problem domain and hence the state space becomes highdimensional\ua0involving both continuous and discrete variables with complex relations. We study how these can be encoded in\ua0a suitable manner using probabilistic generative models so that robots can achieve stable and robust goal-directed grasps by\ua0exploiting feedback loops from multisensory data