Joint action recognition and pose estimation from video

Action recognition and pose estimation from video are closely related tasks for understanding human motion, most methods, however, learn separate models and combine them sequentially. In this paper, we propose a framework to in-tegrate training and testing of the two tasks. A spatial-temporal And-Or graph model is introduced to represent ac-tion at three scales. Specifically the action is decomposed into poses which are further divided to mid-level ST-parts and then parts. The hierarchical structure of our model captures the geometric and appearance variations of pose at each frame and lateral connections between ST-parts at adjacent frames capture the action-specific motion informa-tion. The model parameters for three scales are learned dis-criminatively, and action labels and poses are efficiently in-ferred by dynamic programming. Experiments demonstrate that our approach achieves state-of-art accuracy in action recognition while also improving pose estimation