Pose estimation for mobile devices and augmented reality

In this thesis we introduce the reader to the field of Augmented Reality (AR) and describe aspects of an AR system. We show the current uses in treatment of phobias, games, sports and industry. We present the challenges for Optical See-Through Augmented Reality in which the real world is perceived normally by the user and is augmented with virtual objects by means of two displays and two half-translucent mirrors. Since the user does not perceive the world through camera images, as in Video See-Through Augmented Reality, the requirements for accurate alignment between the real and virtual worlds are more strict. Based on the design requirements for optical see through augmented reality, a systemarchitecture for the full AR system is proposed. A pose (position and orientation) estimation architecture is introduced, which separates an application that needs an estimate of a pose, from the sensors that provide partial measurements for this pose. It is a modular architecture in which modules can publish “magazines” to which other modules can subscribe. A magazine is a data stream of which issues can be read concurrently by multiple subscribers. The read-out rate may be lower than the publishing frequency. Each issue of a magazine is a time stamped data package from a stream, such as an image or measurement. The core of the work addresses the largest challenge in optical see-through AR: real-time pose estimation of the user’s eyes by fusing information from various sensors. Image processing techniques and sensor data fusion filters were developed to provide the most accurate estimation of the pose of a user’s head. The system is general enough to be used in other less demanding applications that need an estimate of a pose, such as free roaming automated vehicles in industrial settings. We explored image processing techniques for determining the pose of the camera from a single image of a marker. A marker is presented that minimizes the impact on the environment. Starting from well-known methods to detect edges and corners we developed our own corner detector that is accurate, precise and robust to noise. We presented a method to estimate the camera’s pose from four corners, and evaluated the accuracy in practical experiments. A Kalman filter is constructed and presented in detail that optimally combines the data from various sensors with different update rates, delays and accuracies. We also propose a pluggable Kalman filter set-up that enables sensors to be added and removed easily without changing the central filter that communicates with the application. This facilitates the separation between the sensor modules, the central filter and the application. A prototype AR system was built and evaluated. We present the practical aspect of integrating the sensors and pose estimation methods into a working augmented reality system. Using a SCARA robot to move our set-up, we determined practical accuracies for our system. We showed that one small marker is in general not enough for a full immersive augmented reality experience. We propose some solutions to increase the accuracy of the system and finally we show how we made convincing Augmented Reality demonstrations in our standing cooperation with the AR-lab of the Royal Academy of Arts in The Hague