From visual saliency to video behaviour understanding

In a world of ever increasing amounts of video data, we are forced to abandon traditional methods of scene interpretation by fully manual means. Dealing with such large amounts of data is a non-trivial task that requires efficient selective extraction of parts of a scene which have the potential to develop a higher semantic meaning, alone, or in combination with others. Modelling these dependencies, whilst eliminating as much redundancy from the feature extraction process as possible are the challenges addressed and discussed in this thesis. In the first part, finding the right mechanism to extract and represent meaningful features from dynamic scenes with no prior knowledge is investigated. The novelty of the work is that it is able to select salient scales in both space and time in which a particular spatio-temporal volume is considered interesting relative to the rest of the scene. By quantifying the temporal saliency values of regions of motion, it is possible to consider their importance in terms of both the long and short-term. Temporal saliency is quantified using a measure based on the entropy of the intensity distribution in a spatio-temporal volume and its variation over scales. Results show that it is possible to extract both locally salient features as well as globally salient temporal features from contrasting scenerios. Then focus shifts towards binding these spatio-temporally salient features together so that some semantic meaning can be inferred from their interaction. Firstly, eo- occurrences are accumulated for spatiotemporally proximate salient features to form a local representation. Then at the next level, the co-occurrence of these locally spatio-temporally bound features are accumulated again in order to discover unusual behaviour in the scene. The novelty of this work is that there are no assumptions made about the scene topology or whether interacting regions should be spatially proximate. Results show that it is possible to detect unusual interactions between regions of motion, which can visually infer higher levels of semantics. In the final part, investigation of human behaviour is addressed through classification and detection of interactions between 2 human subjects. Spatio- temporally salient features are grouped to represent people before a loose pose distribution model is extracted for each person by finding the main dyadic relations between the pose co-occurrence distributions of two interacting people using canonical correlation analysis. Levenshtein distance is then used to categorise the features.EThOS - Electronic Theses Online ServiceGBUnited Kingdo