Reference to Objects in Longitudinal Parent-Child Interaction

A cognitive model of language learning needs to be dialogue-driven and multimodal to reflect how parent and child interact, using words, eye gaze, and object manipulation. In this paper, we present a scheme for multimodal annotation of parent-child interaction. We use this annotation for studying invariance across modalities. Our basic hypothesis is that perception of invariance (or synchrony) in multimodal patterns in auditory-visual speech is the device primarily used to reduce complexity in language learning. To this end, we have added verbal and non-verbal annotation to a corpus of longitudinal video and sound recordings of parent-child dyads. We use this data to try to determine if the amount of synchrony across modalities of parent-child interaction decreases as the child grows older and learns more language and gestures.Modelling the emergence of linguistic structures in early childhoo