Integrating knowledge of hands and objects into egocentric action recognition

This thesis primarily focuses on video-based action recognition, emphasising the role of active objects in the process. To begin, the thesis contributes to the extension of a large-scale egocentric (first-person perspective) action recognition dataset specifically tailored to kitchen environments. Additionally, this dataset is associated with annotation labels of objects and hands, enabling in-depth exploration of object interactions in action recognition.Building on this large dataset, the thesis introduces the Interaction Unit Network (IRN), an innovative approach that models interactions between detected objects and hands to facilitate more discriminative and meaningful feature extraction. Experimental results indicate that object-hand interactions significantly contribute to action recognition, underscoring the value of object-centric approaches.Moreover, conventional object detections struggle to capture object deformations (e.g., a carrot being diced) and occlusions or overlapping objects, which can hinder action recognition accuracy. To address these challenges, this thesis presents and contributes to the EPIC-KITCHENS VISOR, a novel pixel-level segmentation dataset for active objects and interacting hands. This dataset not only accurately tracks object movements and transformations but also precisely segments partially occluded or overlapping objects, providing a more comprehensive scene representation. The thesis delves into the identification of active objects and explores long-term object tracking under a new challenge called ``Where did this come from?''.Leveraging the strengths of EPIC-KITCHENS VISOR, the thesis further investigates the potential of action recognition through semantic segmentation inputs. It examines whether recognising the semantics and relative positions of action-relevant objects, as depicted by semantic segmentations, is sufficient for identifying actions in egocentric videos. Additionally, the thesis explores the complementary nature of active object segmentations in enhancing egocentric action recognition performance. Finally, the thesis compares the advantages and disadvantages of optical flow and semantic segmentation as another modality in egocentric action recognition.Through the proposal of cutting-edge methodologies and datasets, this thesis significantly contributes to a more profound understanding of egocentric action recognition from the perspective of active objects, laying a solid foundation for future research and advancements in this field