Pixel level image understanding with deep learning

Ph.D.Pixel-level image understanding and synthesis are very important problems in computer vision since they provide the most comprehensive and detailed understanding of the visual world. In this thesis, we study these problems in RGB image domain, semantic domain and geometric domain.The first problem we address is, how to extract pixel-level semantic knowledge from RGB images. We address it by developing an object clique potential for semantic segmentation. Our object clique potential addresses the misclassified object-part issues arising in solutions based on fully-convolutional networks. Our object clique set, compared to that yielded from segmentproposal based approaches, is with a significantly smaller size, making our method consume notably less computation. Regarding system design and model formation, our object clique potential can be regarded as a functional complement to local-appearance-based CRF models and works in synergy with these effective approaches for further performance improvement.However, training neural network for pixel-level semantic understanding is data hungry. So the second problem we tackle is to learn pixel-level semantic information with only image-level supervision. Our method unifies semantic segmentation and object localization with important proposal aggregation and selection modules. They greatly reduce the notorious error accumulation problem that commonly arises in weakly supervised learning. Our proposed training algorithm progressively improves segmentation performance with augmented feedback in iterations.When human infer the semantics of the visual world, we not only use the RGB information, but also take the geometry into consideration. Thus, the third problem we handle is to incorporate geometry data for semantic segmentation. To be more specific, we study RGBD semantic segmentation. RGBD semantic segmentation requires joint reasoning about 2D appearance and 3D geometric information. We propose a 3D graph neural network (3DGNN) that builds a k-nearest neighbor graph on top of 3D point cloud. Each node in the graph corresponds to a set of points and is associated with a hidden representation vector initialized with an appearance feature extracted by a unary CNN from 2D images. Relying on recurrent functions, every node dynamically updates its hidden representation based on the current status and incoming messages from its neighbors. This propagation model is unrolled for a certain number of time steps and the finial per-node representation is used for predicting the semantic class of each pixel. We use back-propagation through time to train the model.The fourth problem we study is to extract geometry information from RGB images to gain more complete understanding of our visual world. we propose Geometric Neural Network (GeoNet) to jointly predict depth and surface normal maps from a single image. Building on top of two-stream CNNs, our GeoNet incorporates geometric relationships between depth and surface normal via two novel networks, i.e., depth-to-normal and normal-todepth. Depth-to-normal network exploits the least square solution of surface normal from depth and further improves the quality by a residual module. Normal-to-depth network refines the initial depth map based on the constraints induced by surface normal through a kernel regression module which has no learnable parameters. These two networks enforce the underlying model to efficiently predict depth and surface normal which are not only consistent to each other but also more accurate individually. Our GeoNet is able to predict geometrically consistent depth and surface normal maps.Our last work aims to recreate the visual world based on semantic information. Specially, we study synthesizing images from given semantic layouts. We proposed a semi-parametric approach to image synthesis from semantic layouts. The approach combines the complementary strengths of parametric and nonparametric techniques. The nonparametric component is a memory bank of image segments constructed from a training set of images. Given a novel semantic layout at test time, the memory bank is used to retrieve photographic references that are provided as source material to a deep network. The synthesis is performed by a deep network that draws on the provided photographic material. Our approach yields considerably more realistic images than recent purely parametric techniques.在计算机视觉中，像素级別的圈像理解是非常重要得问题。它提供了对于视觉世界最全面和最精细的理解。在这个论文中，我们主要探索了像素级别的图像理解在彩色图像空间，语义全间，几何空间的问题。我们解决的第一个问题是如何从彩色图像中得到像素级别的语义信息。我们提出了基于物体的能量函数。我们的能量函数可以解决全卷机神经网络中物体局部经常被错误分类的问题。与完全基于物体的方法相比，我们的能量函数可以用更少的计算量。在系统方面，我们的能量函数可以看作是基于局部特征的随机场方法的补充，并且能与基于局部特征的方案协同作用，达到更好的效果。但是，训练深度全卷机网络需要大量的数据。{象素级别上作大规模标注要耗费大量的时间。因此为了提高像素级别语义认知的可扩展性，我们研究的第二个问题是如何仅仅依赖图像级别的标注来学习像素级别的信息。我们提出了物体整合模块和物体选择模块，这两个模块无缝地整合了图像像素级别分割和物体定位。他何J大大减少了在弱监督学习中错误累积的问题。我打1提出的训练算法可以逐步提升图像分割的结果。当我们在分析我们的视觉世界的语义信息的时候，我们不仅考虑图像的色彩信息，而且也要利用到物体的几何特征。因此在第三个问题中，我们丁研究了如何结合几何信息，来实现更好的图像分割。更加精确的，我们研究了结合深度信息的图像分割问题。在这个工作中，我们提出了基于三维图模型的神经网络，该网络首先利用三维点的坐标信息，建立了k 近邻图。在这个图上，每一个节点都是点云上的一个点，并且他的初始化特征是由一个二维的卷积神经网络得到的。其特征会根据当前的状态和他临近点的当前状态，进行迭代更新。最后的预测在最终得到的特征空间进行。我们研究的第四个问题是如何从图像中学习几何的信息。我们提出了几何神经网络，同时对深度和法向量进行预测。在两支神经网络分别进行深度和法向量预测的基础上，我们丁的几何神经网络将深度和法向量的几何关系融合在学习当中。深度到法向量的转化利用了三维空间的平面拟合原理。法向量到深度的模块利用周围点的深度和法向量来约束当前点的深度值。几何神经网络将深度和法向量的几何关系深度融合到整个网络学习中，可以使预测得到的深度和法向量更加符合结合关惊。我们最后一个工作研究的是根据语义信息来重新创造自然世界。我打提出了半参数化的方法。该方法结合了全参数方案和非参数方案的优势。非参数部分包含的是来自自然图像的块，我们丁称之为记记池。在测试阶段，对于每一个语义块，我们首先从记忆池中搜索其对应的自然图像块，然后用该自然图像块作为参考，进行进一步图像合成。Qi, Xiaojuan.Thesis Ph.D. Chinese University of Hong Kong 2018.Includes bibliographical references (leaves 110-124).Abstracts also in Chinese.Title from PDF title page (viewed on …)