Monocular Multiview Object Tracking with 3D Aspect Parts

Abstract. In this work, we focus on the problem of tracking objects un-der significant viewpoint variations, which poses a big challenge to tradi-tional object tracking methods. We propose a novel method to track an object and estimate its continuous pose and part locations under severe viewpoint change. In order to handle the change in topological appear-ance introduced by viewpoint transformations, we represent objects with 3D aspect parts and model the relationship between viewpoint and 3D aspect parts in a part-based particle filtering framework. Moreover, we show that instance-level online-learned part appearance can be incorpo-rated into our model, which makes it more robust in difficult scenarios with occlusions. Experiments are conducted on a new dataset of chal-lenging YouTube videos and a subset of the KITTI dataset [14] that include significant viewpoint variations, as well as a standard sequence for car tracking. We demonstrate that our method is able to track the 3D aspect parts and the viewpoint of objects accurately despite significant changes in viewpoint