Experiments on an RGB-D Wearable Vision System for Egocentric Activity Recognition

tech.cornell.edu This work describes and explores novel steps towards activity recognition from an egocentric point of view. Ac-tivity recognition is a broadly studied topic in computer vi-sion, but the unique characteristics of wearable vision sys-tems present new challenges and opportunities. We eval-uate a challenging new publicly available dataset that in-cludes trajectories of different users across two indoor en-vironments performing a set of more than 20 different ac-tivities. The visual features studied include compact and global image descriptors, including GIST and a novel skin segmentation based histogram signature, and state-of-the art image representations for recognition, including Bag of SIFT words and Convolutional Neural Network (CNN) based features. Our experiments show that simple and com-pact features provide reasonable accuracy to obtain ba-sic activity information (in our case, manipulation vs. non-manipulation). However, for finer grained categories CNN-based features provide the most promising results. Future steps include integration of depth information with these features and temporal consistency into the pipeline. 1