A New Approach for Combining Time-of-Flight and RGB Cameras Based on Depth-Dependent Planar Projective Transformations

Image registration for sensor fusion is a valuable technique to acquire 3D and colour information for a scene. Nevertheless, this process normally relies on feature-matching techniques, which is a drawback for combining sensors that are not able to deliver common features. The combination of ToF and RGB cameras is an instance that problem. Typically, the fusion of these sensors is based on the extrinsic parameter computation of the coordinate transformation between the two cameras. This leads to a loss of colour information because of the low resolution of the ToF camera, and sophisticated algorithms are required to minimize this issue. This work proposes a method for sensor registration with non-common features and that avoids the loss of colour information. The depth information is used as a virtual feature for estimating a depth-dependent homography lookup table (Hlut). The homographies are computed within sets of ground control points of 104 images. Since the distance from the control points to the ToF camera are known, the working distance of each element on the Hlut is estimated. Finally, two series of experimental tests have been carried out in order to validate the capabilities of the proposed method.The authors acknowledge funding from the European commission in the 7th Framework Programme (CROPS Grant Agreement No. 246252) and partial funding under ROBOCITY2030-III-CM project (Robótica aplicada a la mejora de la calidad de vida de los ciudadanos. Fase III; S2013/MIT-2748), funded by Programa de Actividades I+D en la Comunidad de Madrid and cofunded by Structural Funds of the EU. Héctor Montes also acknowledges support from Universidad Tecnológica de Panamá. We acknowledge support by the CSIC Open Access Publication Initiative through its Unit of Information Resources for Research (URICI).CHF 1,620 APC fee funded by the EC FP7 Post-Grant Open Access PilotPeer reviewe