Robust Augmented Reality

The field of Computer Vision is concerned with problems that involve interfacing computers with their surrounding environment through cameras. Consequently artificial vision systems can replace human perception in many tasks. Recent advances in technology, such as increase in computational power, good quality low cost CMOS cameras, improvement in peripherals and decreasing form-factor, allow the vision systems to be carried on roaming platforms such as tablet PCs and mobile phones. More generally, it leads to the possibility of wearable visual computing that can assist its carrier such as humans or robots in executing various perception-action tasks. Augmented Reality (AR) technologies already have a history in the field of computer vision and many attempts have been made to use AR to create meaningful, immersive experiences incorporating humans and computers. It is appealing for many applications such as in entertainment and gaming to improve and enrich perception, cognition and interaction by providing extra information and guidance that is not available in the immediate surroundings. This thesis presents the application of computer vision algorithms to create a marker-less, mobile and wearable AR system that can provide assistive services to its users. The overall system design includes definition and implementation of the necessary software modules as well as its combination with digital hardware. In the wearable mobile set-up we discuss in this thesis, the user can perceive 3D virtual moving objects augmenting the real world perceived at the same time. The Delft University of Technology initiated their AR research in 1999 with outdoor head-mounted optical see-through AR, fusing data from a GPS, a natural feature tracking camera, and an inertia tracker, using a desktop PC in a backpack. Soon a switch back was made to indoor AR based on markers and inertia tracker data in order to improve the real-time performance and static and dynamic accuracy of the head-pose estimation by the visual odometry system. For applications we collaborated since 2006 with the Royal Academy of Art in The Hague, which has implemented many projects in art and design with our systems. In this thesis, our focus has shifted from art and design onto the design and implementation of a system for multi-user collaboration and spatial analysis using multiple AR systems; more specifically we selected Crime Scene Investigation (CSI) as our application domain. For this we collaborated with the Systems Engineering Section (SES-TPM) of TU Delft and the National Forensic Institute in The Hague. We also switched from optical see-through head-mounted displays (HMD) to cheaper video see-through head mounted displays a.k.a eye wear. In this thesis, we introduce the challenges of mobile AR and CSI, and derive the system requirements that can meet these challenges. We further discuss how a remote collaborator can work with on-site users by decoupling him/her self from the on-site user's view while assisting the investigation. Then we present the real-time and on-line modules that satisfy these requirements: -Robust, marker-less, extensible auto-motion tracking based on the tracking of 3D key-points observed by a low cost stereo camera pair -Coarse 3D map-building to be able to augment a real scene with virtual objects, while roaming around in e.g. crime scenes -On-line and on-site scene structure capturing by reconstructing a metric, dense 3D map of the scene in real time, with the aim to let off-scene experts guide the CSIs -Human-Computer Interaction (HCI) software that exploits the user's hand motions. This acts as an interaction device for user interface operation instead of other auxiliary equipment, such as keyboard and mouse. With this HCI the user - e.g. a CSI - can place virtual objects in the scene, such as tags indicating the position of found evidence -Software for remote connection to and on-line collaboration with off-scene experts, a lightweight and affordable HMD and a wearable computer The realized head-mounted AR system allows interaction and collaboration between two or more parties and their environment, while it provides tools, guidance and information to on-site and off-site users to perform their tasks both independently and in collaboration with each other. According to our knowledge, this is one of the first examples of a complete 3D stereo AR system that integrates 3D marker-less AR capabilities with dense reconstruction, remote collaboration and HCI in a carefully engineered way that can be applied in the CSI domain and many other applications in which on-scene and off-scene experts work together