Real-time Virtual Object Insertion

We present a system to insert virtual objects into real image sequences in real time. The system consists of off-the-shelf hardware (a camera connected to a Pentium PC) and software to (a) automatically select and track region features despite changes in illumination, (b) estimate three-dimensional position and orientation of surface patches rel-ative to an inertial reference frame despite individual point-features appearing and disappearing, (c) insert a texture-mapped virtual object into the scene so as to make it appear to be part of the scene and moving with it (see Figures 1 and 2). This is all done in real time. Additional graphic Figure 1. An object is placed on a turntable, which is rotated in front of the camera. Four snapshots are shown. features, such as cast shadows, inter-reections, occlusions, can be handled off-line. While virtual object insertion has been demonstrated before, and is even part of commercial video editing products nowadays [1], current systems re-quire several batch steps, including feature tracking, outlier rejection, epipolar bundle adjustment, some of which in-volve human intervention. What our demo shows is that results of only slightly inferior quality can be obtained in a completely automatic fashion (no human intervention what-soever), while processing the sequence causally (the pose of the virtual object at time is decided based on observations of the sequence up to time). This enables a whole ne