Vision and acceleration modalities: Partners for recognizing complex activities

Wearable devices have been used widely for human activity recognition in the field of pervasive computing. One big area of in this research is the recognition of activities of daily living where especially inertial and interaction sensors like RFID tags and scanners have been used. An issue that may arise when using interaction sensors is a lack of certainty. A positive signal from an interaction sensor is not necessarily caused by a performed activity e.g. when an object is only touched but no interaction occurred afterwards. Especially in health care and nursing scenarios this verification may be critical. In our work, we aim to overcome this limitation and present a multi-modal egocentric-based activity recognition approach which is able to recognize the critical objects. As it is unfeasible to expect always a high quality camera view, we enrich the vision features with inertial sensor data that represents the users' arm movement. This enables us to compensate the weaknesses of the respective sensors. We present our results of combining inertial and video features to recognize human activities on different types of scenarios where we achieve an F1-measure up to 79.6%