Object Modeling and Interactive Perception for Robot Manipulation

A key challenge in robotic systems is how to interpret all the data coming from the sensors on-board the system, and how to select the right environment representation that is beneficial for the task at hand. These challenges are particularly difficult for robots that operate in cluttered environments since objects are ambiguous and tough to distinguish in clutter. Moreover, if a robot needs to physically interact with objects, its internal representation needs to be informative enough to perform the intended interaction, and flexible enough to handle the consequences of its actions. One common application that captures these challenges is object finding in clutter, which in addition to passive visual inspection, often requires interaction to reveal the hidden parts of the environment. To interact, robots need to have some knowledge about the target object, e.g. an object model, and need an environment representation that, together with proprioceptive sensing, contains sufficient information to complete a task successfully. This thesis aims to develop an interactive perception framework for robots. Towards this aim the focus is on (i) how to generate and update object models and (ii) how to develop a flexible planning framework for object interaction. The first part of this thesis presents a dataset of household objects and box scenes containing those objects (CLUBS), commonly found in warehouses. The process of data collection with multiple RGB-D cameras mounted on a robotic arm, and accurate object model generation is described. An image annotation pipeline, and object 3D bounding box estimation approach is also proposed. The box scenes within the dataset contain up to 40 household objects in different configurations, and with different levels of clutter. The variability of the provided data is beneficial for evaluating tasks such as object detection, segmentation, reconstruction, and object completion. Since these scenes also contain 2D object bounding boxes and per frame pixel-wise labels, and 3D object bounding boxes in a world frame, they are valuable for the training of learning-based algorithms. The data, as well as all the tools to manipulate it, are made publicly available. The second part of this thesis proposes a system for 3D perception with a twofold objective. The first objective is to provide object detections in cluttered environments for the target objects that are given in the database. The second objective is to incrementally build a database of object models using an RGB-D camera with or without any prior knowledge about the objects in the environment. The system uses a Truncated Signed Distance Function (TSDF) volumetric representation for storing geometry and object instance information obtained from a depth instance segmentation algorithm. The evaluation on different datasets, including the CLUBS dataset, shows that the system is able to detect objects in different levels of clutter, and generate appealing object models. Even though the object models generated with the incremental database are less accurate and less complete than the ones obtained from the dataset, they are still a faithful representation of the actual object and are valuable for further reasoning about the scene. In the last part of the thesis, a Reinforcement Learning (RL) approach is proposed for active and interactive perception-based object search in clutter. If the object detector does not find the target object in a scene even after the camera moves, it might imply that the object is occluded by the other parts of the environment, or not present at all. To address such scenarios, the robot needs to be able to interact with the environment and remove some of the occlusions, therefore a system capable of planning such actions is proposed. An experimental evaluation of the framework on both a simulated and a real world robotic system shows that the proposed system, trained in simulation only, can solve the task efficiently and with a high success rate. Finally, the performance is compared to baseline approaches showing that the proposed method is more time efficient with comparable success rates. The two main components of this thesis addressed the issues of 3D perception in clutter and action selection for object finding. The results have shown that with such a system the robot is able to discover new objects in the environment and complete the ones already in the database. This can be done online and without any prior knowledge, making the approach very general and flexible. In addition to just moving the camera, this thesis proposed an approach that enables robots to plan actions to physically interact with the to environment with the goal of revealing the hidden parts. By deploying such a framework on a real robot, it allows the robot to understand the environment on an object-level and based on such a representation, plan meaningful actions for a given task, e.g. object finding in clutter