Towards multimodal activity recognition in complex scenarios

Activity recognition deals with the task of figuring out a person’s current activity based on the reading of sensors. Many downstream applications can utilize this information to aid potential users. Fitness tracking devices, for instance, try to determine the timespan a user was running or riding a bike to aid workout goals. Other applications can be found in the health sector, where activity recognition can be used to gain insights about patients that have cognitive disabilities and may forget important daily activities like food or medicine consumption. It can also aid logistics by streamlining processes, e.g. in order picking scenarios for warehouses. For many of these applications, either one or multiple sensors are utilized. In this work, we look at multimodal activity recognition, meaning we consider multiple sensors of different types. We believe that the combination of different modalities can improve the overall accuracy of current methods. Since sensors are becoming cheaper and smaller, practical implementation of multi-sensor systems become more feasible. Specifically, we use video and inertial information for our methods. With this setting in mind, our contribution is spread throughout the whole activity recognition pipeline. We create datasets, develop new annotation methods, and evaluate new types of features and models for activity recognition in both industrial and personal use