Shadow detection and removal with deep learning

Ph.D.Shadows are light-deficient regions where the lights are occluded by the associated objects. On the one hand, shadow provides visual cues for perceiving lighting directions, object shapes, and scene geometry. On the other hand, the presence of shadows could degrade the performance of high-level computer vision tasks such as object detection and tracking. In this thesis, we explore a series of aspects of shadow detection and removal. First, we present a novel deep neural network design that analyzes the spatial image context in a direction-aware manner for shadow detection and removal, since shadows are easily recognized by comparing with the surroundings and the image context in different directions would give different contributions to indicate shadows. Second, we prepare a new benchmark dataset with shadow images and labeled shadow masks to support shadow detection in the complex world and design a fast shadow detection network to detect shadows with real-time performance. Third, we formulate a deep framework, named as Mask-ShadowGAN, which learns to remove shadows from unpaired training data by automatically learning to produce shadow masks from shadow images, using the learned masks to guide shadow generation from shadow-free images, and optimizing shadow generation and shadow removal networks through re-formulated cycle-consistency constraints. Last, we formulate an instance shadow detection task that aims to find shadow instances paired with object instances, prepare a shadow-object association dataset with labeled shadow and object instances, and design an end-to-end deep neural network to predict shadow and object instances, shadow-object associations, and light directions. In all these works, our methods achieve state-of-the-art performance against the current methods, and we further demonstrate the applicability of our shadow detection results on several tasks, such as object detection, light direction estimation, and photo editing.陰影是由於一些物體遮擋了光源而造成的光線不足的區域。一方面，陰影提供了視覺信息，用於感知圖像中光照的方向、物體的形狀和場景的幾何特徵。另一方面，陰影的存在會降低高級計算機視覺任務的性能,例如目標的檢測和跟踪。在本文中，我們探討了關於陰影檢測和去除的一系列問題。首先，我們提出了一種新的深度神經網絡結構，該網絡可以方向性的分析圖像的空間上下文信息來檢測和去除陰影，因為藉助於周圍的環境信息有助於識別陰影，此外不同方向的圖像空間上下文信息為推斷陰影區域提供了不同的貢獻。其次，我們準備了一個有陰影圖像和標註的陰影掩膜的新的基準數據集，用來支持複雜環境中的陰影檢測任務，並設計一個快速的陰影檢測網絡來實時地檢測陰影。此外，我們開發了一個名為Mask-ShadowGAN 的深度框架從未匹配的訓練數據中學習去除陰影，該框架通過自動的從陰影圖像中學習生成陰影掩膜，使用生成的掩膜引導無陰影的圖像生成陰影並通過重新制定的循環一致性約束條件來優化陰影生成和陰影去除網絡。最後，我們制定了一個實例陰影檢測任務，該任務旨在查找配對的陰影實例和目標實例。同時，我們準備了帶有標籤的陰影實例、目標實例以及陰影-目標關聯的數據集，設計了端到端的深度神經網絡來預測陰影實例、目標實例、陰影-目標關聯和光源方向。在所有工作中，我們的方法相對於當前方法均具有最先進的性能，並且我們進一步證明了陰影檢測結果在諸如目標檢測，光源方向估計和照片編輯等多項任務上的適用性。Hu, Xiaowei.Thesis Ph.D. Chinese University of Hong Kong 2020.Includes bibliographical references (leaves 80-90).Abstracts also in Chinese.Title from PDF title page (viewed on March 15, 2022)