Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations

Human action recognition from videos is a chal-lenging machine vision task with multiple im-portant application domains, such as human-robot/machine interaction, interactive entertain-ment, multimedia information retrieval, and surveillance. In this paper, we present a novel ap-proach to human action recognition from 3D skele-ton sequences extracted from depth data. We use the covariance matrix for skeleton joint locations over time as a discriminative descriptor for a se-quence. To encode the relationship between joint movement and time, we deploy multiple covari-ance matrices over sub-sequences in a hierarchical fashion. The descriptor has a fixed length that is independent from the length of the described se-quence. Our experiments show that using the co-variance descriptor with an off-the-shelf classifica-tion algorithm outperforms the state of the art in ac-tion recognition on multiple datasets, captured ei-ther via a Kinect-type sensor or a sophisticated mo-tion capture system. We also include an evaluation on a novel large dataset using our own annotation.