Continuous and Discontinuous Warping-based Video Motion Correction

影片因為其豐富的內容呈現總是令人感到驚奇與興奮，特別是影 片中的動作內容。有許多影片處理的相關技巧跟應用也因此被設計出 來，而其中一項就是影片動作校正(video motion correction)。影片動作 校正意味著經由改變、消除或保留影片中的動作來產生各種應用與效 果進而取得比較好的影片品質和使用者經驗。 大部分影片動作校正應用都遵循一個三步驟的框架：動作估計、動 作補償與影像合成。依照各種應用不一樣的需求與目標，動作估計可 以分為密集與稀疏動作估計，動作補償可以分為物件與攝影機動作補 償而影像合成也可以分為連續與非連續性變形影像合成。影像合成與 動作補償高度相關。當進行動作補償時，應該要同時考慮影像合成結 果的各種品質: 包含影像扭曲、影像完整性與資訊遺失。而影像合成所 使用的模型與方法也會影響最後動作補償的成果。 影片動作校正有許多常見應用，諸如：影片穩定化、劇場動態圖片 生成與影片比例改變。在影片穩定化裡，我們提出一個可以直接穩定 影片而不需要明確的估計攝影機動作也不用假設動作模型或主動作的 方法。這個方法先從影片取出可靠的特徵軌跡，透過最佳化的方式替 每個影格找到一組變形去平滑這些軌跡進而消除影片的晃動。除此之 外，在最佳化時同時也考慮被穩定影片的品質並選擇一個穩定完後具 有比較少未補完區域的影片。實驗顯示我們的方法可以處理具有近、 遠或多物件的複雜影片。對於劇場動態圖片生成，我們提出一個幫助 使用者容易產生劇場動態圖片的系統。這個提出的系統只要求使用者 選擇一個參考的影格與指定一個要保留動作的大概區域，因此可以大 幅減低使用者的負擔。利用這些簡單的輸入，這個系統經由動作估計 與影像變形自動產生無縫的劇場動態圖片。實驗顯示我們的方法可以 產生視覺上比一般常見的方法還好的結果。最後，我們提出一個基於 圖層的影像合成方法來改變立體圖片比例。我們使用一個自創的方 法在改變比例同時去保留立體視差(可視為物件在兩個不同影格的動 作)。基於圖層的混合式方法是一個非連續變形方法，可以使用在影片 動作校正應用上，並得到不錯的效果。立體圖片比例改變可視為兩影 格影片比例改變的特例，所以我們也將這個方法擴充到一般多影格影 片裡。 我們在不同的應用中基於三步驟的架構提出不同的方法，並得到相 較其他常見方法好的結果。我們相信這些方法也可以用相似的方式擴 充到其他相關的影片動作校正應用。Videos have lots of fun and become so popular because of the richness of its content, especially the motions. There are varieties of techniques and applications designed for processing videos, and one of these techniques can be called as video motion correction. Video motion correction means to process videos by changing, removing or preserving motion in a video to create some applications and effects and get better qualities and user experience. Most of video motion correction applications follow a three-step framework: motion estimation, motion compensation and image composition. Depending on different requirements or goals of applications, motion estimation can be categorized into dense and sparse motion estimation. Motion compensation can be divided into object and camera motion compensation. Image composition can also be divided into continuous and discontinuous warpingbased image composition. Image composition is highly related to the motion compensation step. We should also consider the qualities of the composited result including image distortion, image completeness and information loss while compensating motion. And the chosen model and method used in image composition will affect the results of motion compensation. Several applications can be created based on the video motion correction framework, such as video stabilization, cinemagraph creation, and video resizing. For video stabilization, we propose a method to directly stabilize a video without explicitly estimating camera motion, thus assuming neither motion models nor dominant motion. The method first extracts robust feature trajectories from the input video. Optimization is then performed to find a set of transformations to smooth out these trajectories and stabilize the video. In addition, the optimization also considers quality of the stabilized video and selects a video with not only smooth camera motion but also less unfilled area after stabilization. Experiments show that our method can deal with complicated videos containing near, large and multiple moving objects. As to cinemagraph creation, we propose a system to assist users on creating cinemagraphs from videos with ease. The proposed system relieves users’ burdens by only asking users to select a frame as the reference frame and draw a rough mask to identify the region where motion will be preserve. With these simple inputs, the proposed system automatically produces seamless cinemagraphs by using motion estimation and image warping. Experiments show that our method can produce more visually pleasing results than the popular masking methods. Finally, we propose a layer-based image composition method for stereoscopic image resizing which can also be considered as two-frame video resizing. While resizing, we utilize a novel method to preserve disparities which can be considered as object motion in two-frame video resizing. This proposed hybrid layer-based method is also a discontinuous warping method which can be used in video motion correction applications with better results. And we also extend it to multi-frame video resizing. We propose different methods in different applications of video motion correction based on the three-step framework and get better results than other common methods. We believe that these methods can also be applied on other video motion correction applications with the similar way.致謝i 中文摘要iii Abstract v 1 Video Motion Correction 1 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 A framework of video motion correction . . . . . . . . . . . . . . . . . . 1 1.2.1 Motion estimation . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2.2 Motion compensation . . . . . . . . . . . . . . . . . . . . . . . 3 1.2.3 Image composition . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3.1 Video stabilization . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3.2 Cinegraph creation . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3.3 Video resizing . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2 Application: Video Stabilization 9 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3 Robust feature trajectories . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.4 Video stabilization by optimization . . . . . . . . . . . . . . . . . . . . . 15 2.4.1 Trajectory weights . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.4.2 Objective function . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.4.3 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3 Application: Cinemagraph Creation 23 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.3 Cinemagraph creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.3.1 Simple masking method . . . . . . . . . . . . . . . . . . . . . . 26 3.3.2 Motion estimation and evaluation . . . . . . . . . . . . . . . . . 27 3.3.3 Warping-based video processing . . . . . . . . . . . . . . . . . . 28 3.3.4 Boundary registration . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3.5 Seamless looping . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3.6 Optimal boundary conditions . . . . . . . . . . . . . . . . . . . . 31 3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 vii 3.4.1 Rigid objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.4.2 Non-rigid objects . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.4.3 Complex objects . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4 Application: Stereoscopic Image Resizing 43 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.3.1 Multi-layer image compositing . . . . . . . . . . . . . . . . . . . 46 4.3.2 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . 47 4.3.3 Image quality energy . . . . . . . . . . . . . . . . . . . . . . . . 47 4.3.4 Stereoscopic quality energy . . . . . . . . . . . . . . . . . . . . 50 4.3.5 Importance energy . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.4 Implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.7 Extension: video resizing . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.7.1 Motion regularization energy . . . . . . . . . . . . . . . . . . . . 59 4.7.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.7.3 Limitation and conclusions . . . . . . . . . . . . . . . . . . . . . 60 5 Conclusions 65 Bibliography 6