Implementation of a mixed-reality flight simulator: blending real and virtual with a video-see-through head-mounted display

Conventional flight simulators usually include a complex and expensive outside vision projection system. Especially scenarios where the helicopter pilots look far to the side or through the windows near the pedals require a large projection dome to provide an image of the outside world. Additionally, simulators used for research need to be highly customizable: For rapid prototyping of new flight deck designs, the cockpit mockup must be adaptable enough to change the appearance and arrangement of its elements. The recent technological advancements of head-mounted displays (HMDs) offer many new ways to create a simulator that fulfills the stated requirements at moderate cost. A non-see-through HMD can immerse the pilots into a computer-generated cockpit with "unrestricted" virtual out-the-window view. The downwards view is even better than with dome projections. Such a fully virtual approach, however, requires complex finger-tracking and haptic feedback solutions to enable the user to interact with the cockpit. By contrast, a video-see-through HMD allows us to selectively combine a highly customizable virtual world with a video-stream of the real surroundings. One can, for instance, show the pilot's hands and relevant parts of the physical flight deck mockup, enriched with virtual elements and virtual out-the-window view. In such a mixed setup, the pilots can naturally and directly interact with conventional input devices in an otherwise virtual environment. The paper presents our implementation of a mixed reality simulator with the Varjo XR-3 video-see-through HMD. We assess different variants, discuss implementation details like real-to-virtual-world-alignment, and explain the major challenges of such setups