Augmented reality for robot development and experimentation

The successful development of autonomous robotic systems requires careful fusion of complex subsystems for perception, planning, and control. Often these subsystems are designed in a modular fashion and tested individually. However, when ultimately combined with other components to form a complete system, unexpected interactions between subsystems can occur that make it difficult to isolate the source of problems. This paper presents a novel paradigm for robot experimentation that enables unified testing of individual subsystems while acting as part of a complete whole made up of both virtual and real components. We exploit the recent advances in speed and accuracy of optical motion capture to localize the robot, track environment objects, and extract extrinsic parameters for moving cameras in real-time. We construct a world model representation that serves as ground truth for both visual and tactile sensors in the environment. From this data, we build spatial and temporal correspondences between virtual elements, such as motion plans, and real artifacts in the scene. The system enables safe, decoupled testing of component algorithms for vision, motion planning and control that would normally have to be tested simultaneously on actual hardware. We show results of successful online applications in the development of an autonomous humanoid robot.