Pixel level scene understanding with segmentation

Ph.D.Pixel level scene understanding is a fundamental while challenging task in computer vision. It predicts dense values for all pixels in the scene, and is regarded as important to help achieve deep understanding of scene, objects, and human. In this thesis, we present our efforts in developing segmentation systems for pixel-wise scene recognition. The series of our work contains high accuracy semantic segmentation architectures, a high efficiency semantic segmentation framework, a unified panoptic segmentation system and a 3D point cloud segmentation structure.First, we build a high accuracy semantic segmentation architecture. We exploit the capability of global context information by different-region-based context aggregation through our pyramid pooling module together with the proposed pyramid scene parsing network (PSPNet). Our global prior representation is effective to produce good quality results on the scene parsing task, while PSPNet provides a superior framework for pixel-level prediction. The proposed approach achieves state-of-the-art performance on various datasets. It came first in ImageNet scene parsing challenge 2016, PASCAL VOC 2012 benchmark and Cityscapes benchmark. A single PSPNet yields the new record of mIoU accuracy 85.4% on PASCAL VOC 2012 and accuracy 80.2% on Cityscapes.Second, we boost the semantic segmentation accuracy with attention models. We notice information flow in convolutional neural networks is restricted inside local neighborhood regions due to the physical design of convolutional filters, which limits the overall understanding of complex scenes. In this work, we propose the point-wise spatial attention network (PSANet) to relax the local neighborhood constraint. Each position on the feature map is connected to all the other ones through a self-adaptively learned attention mask. Moreover, information propagation in bi-direction for scene parsing is enabled. Information at other positions can be collected to help the prediction of the current position and vice versa, information at the current position can be distributed to assist the prediction of other ones. Our proposed approach achieves top performance on various competitive scene parsing datasets, including ADE20K, PASCAL VOC 2012 and Cityscapes, demonstrating its effectiveness and generality.Third, we address the problem of segmentation efficiency that is vital for various real applications. We focus on real-time semantic segmentation in this work. It finds many practical applications and yet is with fundamental difficulty of reducing a large portion of computation for pixel-wise label inference. We propose an image cascade network (ICNet) that incorporates multi-resolution branches under proper label guidance to address this challenge. We provide in-depth analysis of our framework and introduce the cascade feature fusion unit to quickly achieve high-quality segmentation. Our system yields real-time inference on a single GPU card with decent quality results evaluated on challenging datasets like Cityscapes, CamVid and COCO-Stuff.Fourth, to further obtain the knowledge of instance information inside the scene, we then focus our attention on panoptic segmentation and propose the Unified Panoptic Segmentation Network (UPSNet). On top of a single backbone residual network, we first design a deformable convolution based semantic segmentation head and a Mask R-CNN style instance segmentation head which solve these two subtasks simultaneously. More importantly, we introduce a parameter-free panoptic head which solves the panoptic seg-mentation via pixel-wise classification. It first leverages the logits from the previous two heads and then innovatively expands the representation for enabling prediction of an extra unknown class which helps better resolve the conflicts between semantic and instance segmentation. Additionally, it handles the challenge caused by the varying number of instances and permits back propagation to the bottom modules in an end-to-end manner. Extensive experimental results on Cityscapes, COCO and our internal dataset demonstrate that our UPSNet achieves state-of-the-art performance with much faster inference.Finally, beyond the deep understanding of 2D images, we achieve 3D point clouds understanding and propose the novel PointWeb architecture. It is a new approach to extract contextual features from local neighborhood in a point cloud. Unlike previous work, we densely connect each point with every other in a local neighborhood, aiming to specify feature of each point based on the local region characteristics for better representing the region. A novel module, namely Adaptive Feature Adjustment (AFA) module, is presented to find the interaction between points. For each local region, an impact map carrying element-wise impact between point pairs is applied to the feature difference map. Each feature is then pulled or pushed by other features in the same region according to the adaptively learned impact indicators. The adjusted features are well encoded with region information, and thus benefit the point cloud recognition tasks, such as point cloud segmentation and classification. Experimental results show that our model outperforms the state-of-the-arts on both semantic segmentation and shape classification datasets.像素級場景理解是計算機視覺里基礎但賦有挑戰性的任務。它預測場景中所有像素的密集值，並被視為很重要能夠幫助深入了解場景，物體和人。在本論文中，我們展示了基於分割系統去做像素級別場景識別的努力。本系列作品包含高精度的語義分割架構，高效率的語義分割框架，統一的全景分割系統和三維點雲分割結構。首先，我們構建了高精度語義分割架構。基於不同區域的上下文信息聚合，我們提出了金字塔池化模塊和金字塔場景解析網絡（PSPNet），這有助於提升探索全局上下文信息的能力。我們的全局先驗表徵有效地在場景解析任務中產生高質量的結果，並且PSPNet 為像素級預測提供了優越的框架。所提出的方法在各種數據集上實現了最先進的性能。它獲得了2016 年ImageNet場景解析挑戰賽的冠軍，同時在PASCAL VOC 2012 和Cityscapes 數據集中也達到了第一。單個PSPNet 在PASCAL VOC 2012 和Cityscape 數據集上創造了新紀錄，分別獲得了85.4％ mIoU 的準確率和80.2％ mIoU 的準確率。其次，我們用注意力模型進一步提高了語義分割的準確性。我們注意到由於卷積濾波器的物理設計，卷積神經網絡中的信息流被限制在局部鄰域區域內，這限制了對複雜場景的整體理解。在這項工作中，我們提出了逐點空間注意力網絡（PSANet）來緩和局部鄰域約束。特徵圖上每個點都和其他所有點通過自適應學習到的注意力圖相連接。此外，用於場景解析的雙向信息傳播流也被建立。其他點的信息可以被收集起來幫助當前點的預測，反之亦然，當前點的信息也可以被傳播出去幫助其他點的理解。我們提出的方法在各種極具競爭力的場景解析數據集上實現了最佳性能，包括ADE20K，PASCAL VOC 2012 和Cityscapes，這展示了其有效性和通用性。第三，我們解決了對各種實際應用至關重要的分割效率問題。在這個任務中我們專注於實時語義分割。它有許多實際應用，但是由於需要像素級的推斷使得計算量難以得到大幅度的減少。我們提出了一種圖像級聯網絡（ICNet）去應對這一挑戰，它結合了多分辨率的信息和多分支的標籤監督。我們對框架進行了深度分析，並引入級聯特徵融合單元，以實現快速高質量的分割。我們的系統可在單個GPU 卡上實現實時推斷，實驗顯示在多個具有挑戰性的數據集上都可以得到優秀的分割結果。第四，為了進一步獲得場景內實例信息的知識，我們將注意力集中在全景分割上，並提出統一全景分割網絡（UPSNet）。在單個骨幹殘差網絡的基礎上，我們首先設計了一個基於可變形卷積的語義分割模塊和一個Mask R-CNN樣式的實例分割模塊，這樣可以同時解決這兩個子任務。更重要的是，我們引入了一個無參數的全景分割模塊，它通過像素級的分類解決了全景分割。它首先利用前兩個模塊的邏輯值，然後創新地擴展一個額外的類別來預測未知類別，這有助於更好地解決語義和實例分割之間的衝突。另外，它處理了由不同數量的實例而引起的挑戰，並允許以端到端的方式做反向傳播來進行訓練。Cityscapes，COCO 和我們的內部數據集的廣泛實驗結果表明，我們的UPSNet 以更快的推理速度實現了最佳的全景分割精度。最後，除了對二維圖像的深入理解之外，我們也獲取了三維點雲場景的理解並提出了新的PointWeb 架構。這是一種從點雲中的局部鄰域內提取上下文特徵的新方法。與以前的工作不同，我們在局部鄰域中將每個點密切地連接在一起，旨在基於局部區域特徵加強每個點的表達以更好地表示該區域。一種新穎的自適應特徵調整模塊，被用於找到點之間的相互作用。對於每個局部區域，攜帶點對之間影響力的影像圖作用在特徵差異圖上。然後，根據自適應學習到的影響權重，同一區域中的其他特徵對當前特徵產生拉動或推動效果。調整後的特徵能很好地編碼區域信息，因此有益於點雲識別任務，例如點雲分割和分類。實驗結果表明，我們的模型在語義分割和形狀分類數據集上都優於現有技術。Zhao, Hengshuang.Thesis Ph.D. Chinese University of Hong Kong 2019.Includes bibliographical references (leaves 119-131).Abstracts also in Chinese.Title from PDF title page (viewed on 20, May 2021)