Natural action recognition using invariant 3D motion encoding

Abstract. We investigate the recognition of actions “in the wild ” using 3D mo-tion information. The lack of control over (and knowledge of) the camera con-figuration, exacerbates this already challenging task, by introducing systematic projective inconsistencies between 3D motion fields, hugely increasing intra-class variance. By introducing a robust, sequence based, stereo calibration tech-nique, we reduce these inconsistencies from fully projective to a simple similarity transform. We then introduce motion encoding techniques which provide the nec-essary scale invariance, along with additional invariances to changes in camera viewpoint. On the recent Hollywood 3D natural action recognition dataset, we show im-provements of 40 % over previous state-of-the-art techniques based on implicit motion encoding. We also demonstrate that our robust sequence calibration sim-plifies the task of recognising actions, leading to recognition rates 2.5 times those for the same technique without calibration. In addition, the sequence calibrations are made available