1Feature Fusion for Efficient Content-Based Video Retrieval

Abstract—Content-based video retrieval is a complex task because of the large amount of information in single items and because databases of videos can be very large. In this paper we explore a possible solution for efficient similar item retrieval. In our experiments we combine relevant feature sets together with a learned Mahalanobis metric while using an efficient nearest neighbor search algorithm. The efficient nearest neighbor algorithms we compare are Locality Sensitive Hashing and Vantage Point trees. The two options are compared to several baseline systems in the general video retrieval framework. We used three sets of features to test the system: SURF features, color histograms and topics. The topics where extracted using a Latent Dirichlet Allocation topic model. We show that fusing the individual feature sets with a learned metric improves the performance upon the best individual feature set. The feature fusion can be combined with an efficient nearest neighbor search algorithm to reduce the number of exact distance computations with limited impact on retrieval performance. Index Terms—Content-based video retrieval, feature fusion, metric learning, efficient retrieval, nearest neighbor search, locality sensitive hashing, vantage point trees I