EDICS: ARS-SRE AND ARS-IIU 1 Robust Spatio-temporal Matching of Electronic Slides to Presentation Videos

We describe a robust and efficient method for automatically matching and time-aligning electronic slides to videos of corresponding presentations. Matching electronic slides to videos provides new methods for indexing, searching, and browsing videos in distance learning applications. However, robust automatic matching requires deliberate considerationof variouscomplicationsin the video such as varied frame composition, slide distortion, camera movement, low-quality video capture, and arbitrary slides sequence. Our fully-automaticapproach combinesimage-based matching of slide images to video frames with a temporal model for slide changes and camera events. For initial image matching we extract scale invariant feature transformation (SIFT) keypoints from both slide images and video frames, and match them subject to a consistent projective transformation (homography) by using random sample consensus (RANSAC). This yields an initial set of matches that we then use to build an unsupervised background model and a binary classifier that more reliably separates frames with slides from those without. We then propose a new matching scheme for exploiting less distinctive SIFT keypoints that enables us to tackle more difficult images. We make the matchingeven more robust by using matching probabilitiesestimated from visual information as part of a hidden Markov model (HMM) that integrates temporal information and detected camera operations. We report on detailed quantitative experiments that characterize the contribution of each part of our approach and demonstrate an average accuracy of over 95 % in 13 presentation videos with different styles