Regularized Semi-Dense Map Reconstruction from a Monocular Sequence based on Piecewise Planar Constraint and Gauss Newton Method

近幾年來，利用單眼攝影機進行三維環境的重建已成為一個熱門且具有挑戰性的題目。這項技術可以應用在無人載具進行自動導航、環境探勘以及自動避障，除此之外，還可以應用在擴增實境上。由於相機並不配有慣性測量單元，定位相機與三維環境的重建必須同時進行，在此篇論文中，相機的姿態估測使用以影像上特徵點為基礎的方式[24: Lepetit et al. 2009] 以及直接利用影像上像素的方式 [1: Engel et al. 2014]，相機姿態估測需要使用到重建出的半稠密深度圖，而這些點都是影像上梯度較高的點且很容易就變得很雜亂。因此，本篇論文提出了一個可正規化重建出的半稠密地圖而不影響相機估測精確度的方法，該正規化方法可以去除雜訊並且平滑重建出的地圖，除此之外，此正規化方法與兩張影像上的像素亮度值有關，不同於其他方法只基於重建點之深度值以及空間上的關係。地圖的重建可以分成三個部分: 立體匹配法、片狀平面約束以及平面最佳化。由於高梯度的影像部分通常都很狹窄，不容易通過片狀平面的約束，因此提出了一個利用重建高梯度區域周圍低梯度點來增寬高梯度區域的匹配方式。當半稠密的地圖重建完後，片狀平面約束會估測半稠密地圖上片狀的平面，最後透最佳化所有半稠密地圖上的片狀平面。 在本篇論文中，提出的立體匹配法ORB[27: Rublee et al. 2011] 特徵點的預設深度資訊、K維樹 [36: Bentley 1975] 、優先序列以及方向梯度直方圖的亂度構成，目標是透過極平面來正確匹配在兩張影像中圍繞在高梯度點附近的低梯度點，由於低梯度點並沒有紋理上的特徵，很難在兩張影像中做匹配，所以搜尋週圍最適合並且有紋理的區域來做匹配的流程，首先若這個像素並沒有逆深度的假設模型，在此像素點週圍具有深度知訊的ORB 特徵點會用來初始化該像素點的深度值，如此可以縮短極線上的搜尋長度以及增加匹配的精準度，搜尋高梯度紋理區域是利用K維樹進行K最近鄰演算法，並將搜尋到的點依照梯度最大優先的條件存入優先序列中，若搜尋到的點通過立體匹配約束，則這個高梯度點會形成一個5×5 像素大小的模板，然後用來進去立體匹配，當兩張影像上的模板的相差通過隨著梯度直方圖的亂度而改變的閥值時，相對應的點便視為匹配成功。 本篇論文正規化的部分，假設投影到三維空間中的每一小片狀點雲都符合一個平面，每個片狀點雲的大小為5×5 像素，由於這個假設在兩個物體的交接處或不連續的區域並不會成立，此時片狀平面約束便使用來篩選掉非平面的區域，通過片狀平面約束後，利用高斯牛頓法最小化兩張影像上投影自片狀平面區塊的差值以得到最佳化的平面參數，之後，最佳化平面參數用來消除點雲中的雜點以及讓點雲平滑化。實驗結果顯示所提出的正規化演算法可以消除大部分的雜點，以及重建出一個更加清楚的環境。Three-dimensional environment reconstruction from a monocular camera has been a popular and a challenge research topic in past few years. This technique can be applied to unmanned vehicles to perform automatic navigation, environment exploration and automatic obstacle avoidance. In addition, it can also be applied to augmented reality. Since the camera is not equipped with an inertial measurement unit (IMU), it is necessary to locate the camera position and map the environment simultaneously. In this thesis, the camera pose estimation is based on feature based method [24: Lepetit et al. 2009] and direct method [1: Engel et al. 2014]. The camera localization thread is depend on the semi-dense map which is the high gradient area in image and is easily to become noisy. Hence, a method that can regularize the reconstructed semi-dense map without affect the accuracy of the camera pose localization is proposed in this thesis. The regularization method can eliminate the noise and smooth the semi-dense map. Furthermore, the regularization method is related to the photometric information between two images, unlike other methods only using the information of the depth and spatial relation. The reconstruction algorithm can be divide into three parts: stereo matching, piecewise planar constraint, and plane optimization. Since the high gradient areas are always narrow and hard to apply the piecewise planar constraint, a stereo matching method that can broaden the high gradient area by using their nearby low gradient pixels is proposed. After the semi-dense map is reconstructed, the semi-dense map will propagate to the piecewise planar constraint which can estimate the initial piecewise planes for each pixel. Finally, the optimization method is applied to optimize each estimated piecewise plane. In this thesis, the proposed stereo matching is composed of prior depth of ORB feature [27: Rublee et al. 2011], KD-Tree [36: Bentley 1975], Priority Queue and the entropy of the histogram of oriented gradient. The aim is to match the low gradient area around the high gradient area between two images correctly by using the epipolar geometry. It is hard to match two textureless areas between two images, so the best nearby texture area is searched to do the matching procedure. Firstly, if one pixel does not hold an inverse depth hypothesis, the nearby ORB features which has initial depth knowledge is used to initiate the inverse depth value, which can shorter the epipolar line searching length and improve the accuracy of the matching result. Searching the texture area which contains high gradient pixel is done by using k nearest neighbor search with KD-Tree, and sorting the searched pixels in accordance with the gradient magnitude by the priority queue. If the searched point passes the stereo searching constraint, the searched high gradient point will form a 5×5 pixels template and be used to do the stereo line searching. The corresponding points are considered to be matched if the residual between the templates in two image pass the stereo matching threshold which will change with the value of the searching region’s entropy of the histogram of oriented gradient. In the regularization part of this thesis, each tiny piece of point cloud projected from the image in 3D coordinate is assumed to fit a plane. The corresponding size in the image of each piece is set to 5×5 pixels. Since the assumption will not hold if the piece is in the border between two different objects or the discontinuous area, the planar constraint is applied to discriminate the non-planar region. After passing the planar constraint, Gauss-Newton method is used to minimize the photometric error between the two patches which projected from the piece in 3D coordinate in two images and the optimal parameters of the plane can be obtained. Afterwards, the optimal parameters are used to eliminate the noises and smooth the point cloud. The experimental results demonstrate that the proposed regularization algorithm can eliminate most of the noises and reconstruct a more clearly point cloud