Multimodal Dynamic Networks for Gesture Recognition

Multimodal input is a real-world situation in gesture recog-nition applications such as sign language recognition. In this paper, we propose a novel bi-modal (audio and skele-ton joints) dynamic network for gesture recognition. First, state-of-the-art dynamic Deep Belief Networks are deployed to extract high level audio and skeletal joints representation-s. Then, instead of traditional late fusion, we adopt another layer of perceptron for cross modality learning taking the input from each individual net’s penultimate layer. Finally, to account for temporal dynamics, the learned shared repre-sentations are used for estimating the emission probability to infer action sequences. In particular, we demonstrate that multimodal feature learning will extract semantically meaningful shared representations, outperforming individu-al modalities, and the early fusion scheme’s efficacy against the traditional method of late fusion