Coupled Action Recognition and Pose Estimation from Multiple Views

Abstract Action recognition and pose estimation are two closely related topics in understanding human body move-ments; information from one task can be leveraged to as-sist the other, yet the two are often treated separately. We present here a framework for coupled action recognition and pose estimation by formulating pose estimation as an optimization over a set of action-specific manifolds. The framework allows for integration of a 2D appearance-based action recognition system as a prior for 3D pose estima-tion and for refinement of the action labels using relational pose features based on the extracted 3D poses. Our experi-ments show that our pose estimation system is able to esti-mate body poses with high degrees of freedom using very few particles and can achieve state-of-the-art results on the HumanEva-II benchmark. We also thoroughly investigate the impact of pose estimation and action recognition accu-racy on each other on the challenging TUM kitchen dataset. We demonstrate not only the feasibility of using extracted 3D poses for action recognition, but also improved perfor-mance in comparison to action recognition using low-level appearance features