IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING 1 Learning to detect visual grasp affordance

Abstract—Appearance-based estimation of grasp affordances is desirable when 3-D scans become unreliable due to clutter or material properties. We develop a general framework for estimating grasp affordances from 2-D sources, including local texture-like measures as well as object-category measures that capture previously learned grasp strategies. Local approaches to estimating grasp positions have been shown to be effective in real-world scenarios, but are unable to impart object-level biases and can be prone to false positives. We describe how global cues can be used to compute continuous pose estimates and corresponding grasp point locations, using a max-margin optimization for category-level continuous pose regression. We provide a novel dataset to evaluate visual grasp affordance estimation; on this dataset we show that a fused method outperforms either local or global methods alone, and that continuous pose estimation improves over discrete output models. Finally, we demonstrate our autonomous object detection and grasping system on the Willow Garage PR2 robot. Note to Practitioners—Learning grasp affordances for autonomous agents such as personal robots is a challenging task. We propose an unified framework which first detects target objects, infers grasp affordance of the target object, and executes robotic grasp. Our method is mainly based on 2D imagery data which can be more robust when 3D scans are unavailable due to background clutter and material properties such as surface reflectance. One of the future extensions would be to automate the training phase so that robots can actively learn object models by interacting with objects as opposed to having a human in the loop collecting and annotating training images