Learning to reconstruct and segment 3D objects

To endow machines with the ability to perceive the real-world in a three dimensional representation as we do as humans is a fundamental and long-standing topic in Artificial Intelligence. Given different types of visual inputs such as images or point clouds acquired by 2D/3D sensors, one important goal is to understand the geometric structure and semantics of the 3D environment. Traditional approaches usually leverage hand-crafted features to estimate the shape and semantics of objects or scenes. However, they are difficult to generalize to novel objects and scenarios, and struggle to overcome critical issues caused by visual occlusions. By contrast, we aim to understand scenes and the objects within them by learning general and robust representations using deep neural networks, trained on large-scale real-world 3D data. To achieve these aims, this thesis makes three core contributions from object-level 3D shape estimation from single or multiple views to scene-level semantic understanding. In Chapter 3, we start by estimating the full 3D shape of an object from a single image. To recover a dense 3D shape with geometric details, a powerful encoder-decoder architecture together with adversarial learning is proposed to learn feasible geometric priors from large-scale 3D object repositories. In Chapter 4, we build a more general framework to estimate accurate 3D shapes of objects from an arbitrary number of images. By introducing a novel attention-based aggregation module together with a two-stage training algorithm, our framework is able to integrate a variable number of input views, predicting robust and consistent 3D shapes for objects. In Chapter 5, we extend our study to 3D scenes which are generally a complex collection of individual objects. Real-world 3D scenes such as point clouds are usually cluttered, unstructured, occluded, and incomplete. By drawing on previous work in point-based networks, we introduce a brand-new end-to-end pipeline to recognize, detect, and segment all objects simultaneously in a 3D point cloud. Overall, this thesis develops a series of novel data-driven algorithms to allow machines to perceive our real-world 3D environment, arguably pushing the boundaries of Artificial Intelligence and machine understanding.