Learning Basic Object Affordances in a Robotic System

One of the fundamental enabling mechanisms of human and animal intelligence, and equally, one of the great challenges of modern day autonomous robotics is the ability to perceive and exploit environmental affordances. To recognise how you can interact with objects in the world, that is to recognise what they afford you, is to speak the language of cause and effect, and as with most languages, practice is one of the most important paths to understanding. This is clear from early childhood development. Through countless hours of motor babbling, children gain a wealth of experience from basic interactions with the world around them, and from there they are able to learn basic affordances and gradually more complex ones. Implementing such affordance learning capabilities in a robot, however, is no trivial matter. This is an inherently multi-disciplinary challenge, drawing on such fields as autonomous robotics, computer vision, machine learning, artificial intelligence, psychology, neuroscience, and others. In this thesis, we attempt to study the problem of affordance learning by embracing its multi-disciplinary nature. We use a real robotic system to perform experiments using household objects. Camera systems record images and video of these interactions from which computer vision algorithms extract interesting features. These features are used as data for a machine learning algorithm that was inspired in part by ideas from psychology and neuroscience. The learning algorithm is perhaps the main focal point of the work presented here. It is a self-supervised multi-view online learner that dynamically forms categories in one data view, or sensory modality, that are used to drive supervised learning in another. While useful in and of itself, the self-supervised learner can potentially benefit from certain augmentations, particularly over shorter training periods. To this end, we also propose two novel feature relevance determination methods that can be applied to the self-supervised learner. With regard to robotic experiments, we make use of two different robotic setups, each of which involves a robot arm operating in an experimental environment with a flat table surface, with camera systems pointing at the scene. Objects placed in the environment can be manipulated, generally pushed, by the arm, and the camera systems can record image and video data of the interaction. One of the camera systems in one of the setups is a stereo camera, and another in the other setup is an RGB-D sensor, thus allowing for the extraction of range data and 3-D point cloud data. In the thesis, we describe computer vision algorithms for extracting both salient object features from the static images and point cloud data, and effect features from the video data of the object in motion. A series of experiments are described that evaluate the proposed feature relevance algorithms, the self-supervised multi-view learning algorithm, and the application of these to real-world object push affordance learning problems using the robotic setups. Some surprising results emerge from these experiments and as well as those, under the conditions we present, our framework is shown to be able to autonomously discover object affordance categories in data, predict the affordance categories of novel objects and determine the most relevant object properties for discriminating between those categories