Shot-based video retrieval with optical flow tensor and HMMs

Video retrieval and indexing research aims to efficiently and effectively manage very large video databases, e.g., CCTV records, which is a key component in video-based object and event analysis. In this paper, for the purpose of video retrieval, we propose a novel method to represent video data by developing an optical flow tensor (OFT) and incorporating hidden Markov models (HMMs). As video is content-sensitive and normally carries rich motion information of objects, optical flow field is first employed to estimate such motion. Then, a shot HMMs tree is built to model video clips in different levels in a database. Experimental results demonstrate that the newly developed method inherits advantages of both optical flow and HMMs in video representation. With the newly developed video representation, in video retrieval and indexing tasks, no need to exhaustively compare a query video shot with all video shot records in the database. Moreover, the novel representation method works well when linear discriminant analysis (LDA) is utilized to reduce the feature dimensionality and further speed up the retrieval procedure. © 2008 Elsevier B.V. All rights reserved