Mixed Reality Simulation with Physical Mobile Display Devices

This paper presents the design and implementation of a system for simulating mixed reality in setups combining mobile devices and large backdrop displays. With a mixed reality simulator, one can perform usability studies and evaluate mixed reality systems while minimizing confounding variables. This paper describes how mo-bile device AR design factors can be flexibly and systematically explored without sacrificing the touch and direct unobstructed ma-nipulation of a physical personal MR display. First, we describe general principles to consider when implementing a mixed reality simulator, enumerating design factors. Then, we present our imple-mentation which utilizes personal mobile display devices in con-junction with a large surround-view display environment. Stand-ing in the center of the display, a user may direct a mobile device, such as a tablet or head-mounted display, to a portion of the scene, which affords them a potentially annotated view of the area of inter-est. The user may employ gesture or touch screen interaction on a simulated augmented camera feed, as they typically would in video-see-through mixed reality applications. We present calibration and system performance results and illustrate our system’s flexibility by presenting the design of three usability evaluation scenarios