L.: Robust trajectory-space tv-l1 optical flow for non-rigid sequences

Abstract. This paper deals with the problem of computing optical flow between each of the images in a sequence and a reference frame when the camera is view-ing a non-rigid object. We exploit the high correlation between 2D trajectories of different points on the same non-rigid surface by assuming that the displacement sequence of any point can be expressed in a compact way as a linear combination of a low-rank motion basis. This subspace constraint effectively acts as a long term regularization leading to temporally consistent optical flow. We formulate it as a robust soft constraint within a variational framework by penalizing flow fields that lie outside the low-rank manifold. The resulting energy functional in-cludes a quadratic relaxation term that allows to decouple the optimization of the brightness constancy and spatial regularization terms, leading to an efficient opti-mization scheme. We provide a new benchmark dataset, based on motion capture data of a flag waving in the wind, with dense ground truth optical flow for evalua-tion of multi-view optical flow of non-rigid surfaces. Our experiments, show that our proposed approach provides comparable or superior results to state of the art optical flow and dense non-rigid registration algorithms.