Realistic image and video super-resolution with deep learning

Super-resolution (SR) aims at recovering a high-resolution (HR) image or video clip from its corresponding low-resolution (LR) one. Despite that convolutional neural networks (CNN) have recently demonstrated high-quality reconstruction for super-resolution, realistic SR, i.e., recovering natural and realistic textures, remains a challenging problem. The reason is that SR is an intrinsically ill-posed problem. CNN-based methods mitigate this problem by restricting the solution space by priors learned from the external dataset. However, such general priors are not enough for realistic SR. This thesis attempts to address the ill-posed challenge from two aspects: incorporating semantic priors and pushing solutions to a more accurate natural image manifold. Besides the visual-pleasing texture restoration in realistic SR, controllable restoration is also highly desired in real-world scenarios, due to the large variety of user flavors and various application scenarios. We aim to devise a simple yet universal approach to attain a smooth control of diverse imagery effects. In order to transfer what we develop for image realistic SR to videos, we also aim at devising an effective video SR framework that can easily incorporate the above techniques.To constrain the solution space with semantic priors, we propose to explore the semantic segmentation maps as the categorical prior. Our experiments show that segmentation maps encapsulate rich categorical prior up to pixel level. Then we propose a novel Spatial Feature Transform (SFT) layer to allow factorized texture generation in an SR network conditioned on segmentation maps. SFT layers generate affine transformation parameters for spatial-wise feature modulation. SFT layers can be trained end-to-end together with the SR network using the same loss function. During testing, it accepts an input image of arbitrary size and generates a high-resolution image with just a single forward pass conditioned on the categorical priors. Experimental results show that an SR network equipped with SFT can generate more realistic and visually pleasing textures in comparison to state-of-the-art methods.For restricting the solution space into natural image manifold, the Super-Resolution Generative Adversarial Network (SRGAN)[63] is a seminal work. However, the hallucinated details are often accompanied with unpleasant artifacts. To further enhance the visual quality, we thoroughly inspect the details of SRGAN -- network architecture, adversarial loss and perceptual loss, and improve each of them to present an Enhanced SRGAN (ESRGAN). Specifically, we introduce the Residual-in-Residual Dense Block (RRDB) without batch normalization as the basic network building unit. We adopt the idea of letting the discriminator predict relative realness instead of the absolute value from relativistic GAN[52]. Finally, we improve the perceptual loss by using the features before activation, which could provide stronger supervision for brightness consistency and texture recovery. Equipped with these improvements, the proposed ESRGAN achieves consistently better visual quality with more realistic and natural textures than SRGAN and won the first place in the 2018 PIRM-SR Challenge.Instead of learning a deterministic mapping for the desired imagery effect, controllable restoration requires different output effects even the possibility of continuous transition without re-training. We propose a simple yet universal approach to attain a smooth control of diverse imagery effects in many low-level vision tasks, including image restoration, image-to-image translation, and style transfer. Specifically, our method, namely Deep Network Interpolation (DNI), applies linear interpolation in the parameter space of two or more correlated networks. A smooth control of imagery effects can be achieved by tweaking the interpolation coefficients. We also investigate the mechanism of network interpolation from the perspective of learned filters.With the explosion of video applications, it is attractive to extend the realistic SR to videos. An effective video SR framework that can easily incorporate the above techniques is required. Realistic video inputs challenge existing methods from two aspects: 1) how to align multiple frames given large motions, and 2) how to effectively fuse different frames with diverse motion and blur. In this thesis work, we propose a novel Video Restoration framework with Enhanced Deformable convolutions, termed EDVR, to address these challenges. EDVR is a unified framework with unique designs to achieve good alignment and fusion quality in various video restoration tasks. In particular, we devise a PCD alignment module, which aligns features of different frames with deformable convolution in a coarse-to-fine manner. Moreover, we propose a TSA fusion module, which employs attention both temporally and spatially to emphasize informative features for better reconstruction. Thanks to these modules, EDVR not only wins all four tracks in the NTIRE19 Challenges but also demonstrates superior performance to existing methods on several benchmarks of video super-resolution and deblurring.超解析度是指從對應的低解析度圖像或視頻片段中恢復出高解析度的圖像或視頻片段。儘管卷積神經網路（縮寫CNN）已被證明具有高品質重建，但恢復自然且真實的紋理的真實超解析度，仍然是一個具有挑戰性的問題。其原因是超解析度本質上是一個病態問題。基於CNN的方法雖然能夠通過從外部資料集中學習先驗知識來限制超解析度的解空間，從而緩解這一問題，但這些一般的先驗知識對於真實超解析度是遠遠不夠的。本工作試圖從兩個方面來解決這個病態的挑戰：1）結合語義先驗；2）使得解落在更精確的自然圖像流形上。真實超解析度除了要恢復看起來自然的紋理外，可控制恢復在實際場景中也非常需要，因為在實際中，用戶的品味多種多樣，應用場景也不盡相同。我們的目標是要設計一個簡單且通用的方法來實現對不同圖像效果的平滑控制。為了將我們的圖像真實超解析度技術方便地應用到視頻中，我們還致力於設計一個有效的視頻超解析度框架，該框架可以很容易地將上述技術結合進來。為了用語義先驗來約束解空間，我們提出將語義分割圖作為類別先驗。實驗顯示分割圖具有細緻到圖元級的類別先驗。然後，我們提出了一種新的空間特徵變換層（縮寫SFT）來支援基於分割圖的超解析度網路的紋理生成。SFT層會產生仿射變換參數用於具有空間資訊的特徵調製。SFT層可與超解析度網路一起用相同的損失函數進行端到端訓練。在測試過程中，它可接受任意大小的輸入圖像，只需要一次前向傳播就可以根據類別先驗生成一張高解析度圖像。實驗結果表明，與現有方法相比，採用SFT的超解析度網路可生成更真實、自然的紋理。超解析度生成對抗網路（縮寫SRGAN）是將解空間限制到自然圖像流形上的開創工作。然而，其生成的細節往往伴隨著令人不快的瑕疵。為進一步提高視覺品質，我們深入研究了SRGAN的網路結構、對抗性損失和感知損失的細節，並對它們進行了改進，提出了一種增強型SRGAN（縮寫ESRGAN）。具體地說，我們引入了不含批歸一化（縮寫BN）的殘差密集連接模組塊（縮寫RRDB）作為網路構建的基本單元。我們採用了由鑒別器來預測相對真實性的思想，而不是直接使用預測絕對值。最後，我們利用激活前的特徵來改善感知損失，其提供了更強的監督，有利於亮度一致性和紋理恢復。有了這些改進，ESRGAN實現了比SRGAN更真實且自然的紋理生成，並且有更好的視覺品質，並在2018年PIRM-SR挑戰賽中獲得第一名。和以往對所需的圖像效果都有一個確定的輸出不同, 可控的恢復需要在不重新訓練的條件下,輸出不同的效果，甚至是輸出連續過渡效果。我們提出了一種簡單而通用的方法，在許多底層視覺任務中，包括圖像恢復、圖像到圖像的轉換和風格轉換，實現對不同圖像效果的平滑控制。具體地說，我們提出的深度網路插值（縮寫DNI），在兩個或多個相關網路的參數空間中應用線性插值。通過調整插值係數可以實現對圖像效果的平滑控制。我們還從學習到的濾波器的角度研究了網路插值的機理。隨著視頻應用的迅猛發展，將真實超解析度技術擴展到視頻領域具有重要意義。一個有效的視頻超解析度框架可以很容易地將上述技術結合進來。真實的視頻輸入從兩個方面對現有的方法提出了挑戰：1）如何在大運動的情況下對齊多幀輸入；2）如何有效地融合不同運動和模糊下的幀。在這篇論文中，我們基於加強可變形卷積提出了一個新的視頻復原架構，稱為 EDVR。它是一個具有獨特設計的統一框架，可以在各種視頻恢復任務中實現良好的對齊和融合品質。我們設計了PCD模組，以粗略到精細的方式對多幀使用形變卷積進行對齊。此外，我們還提出了一種在空間和時間上都能更好融合資訊特徵的TSA模型。有了這些模組，EDVR不僅在NTIRE19挑戰賽中贏得了所有四個賽道的冠軍，而且在視頻超解析度和去模糊的幾個基準測試中都顯示出優於現有方法的性能。Wang, Xintao.Ph.D. Chinese University of Hong Kong 2020.Includes bibliographical references (leaves 118-138).Abstracts also in Chinese.Title from PDF title page (viewed on October 19, 2022)