Geometric loss functions for camera pose regression with deep learning

Deep learning has shown to be effective for robust and real-time monocular image relocalisation. In particular, PoseNet [22] is a deep convolutional neural network which learns to regress the 6-DOF camera pose from a single im- age. It learns to localize using high level features and is ro- bust to difficult lighting, motion blur and unknown camera intrinsics, where point based SIFT registration fails. How- ever, it is trained using a naive loss function, with hyper- parameters which require expensive tuning. In this paper, we give the problem a more fundamental theoretical treat- ment. We explore a number of novel loss functions for learning camera pose which are based on geometry and scene reprojection error. Additionally we show how to au- tomatically learn an optimal weighting to simultaneously regress position and orientation. By leveraging geometry, we demonstrate that our technique significantly improves PoseNet’s performance across datasets ranging from indoor rooms to a small city