Integrative learning between language and action: A neuro-robotics experiment

Abstract. The current paper introduces a model for associative learn-ing between linguistic modality and behavior modality. The model consti-tutes of language and behavior modules both of which are implemented with a hierarchical dynamic network model and they interact densely through hub-like neurons, the parametric biases (PB). By implementing this model to a humanoid robot in the task of multiple objects manip-ulation, the robot was tutored to associate sentences of two different grammatical types to corresponding sensory-motor schemata. One type is a verb followed by an objective noun as like ”hold red ” or ”hit blue” and the other is a verb followed by an objective noun and further fol-lowed by an adverb phrase as like ”Put red on blue”. Our analysis on the results of a learning experiment showed that two clusters corresponding to these two types of grammatical sentences appear in the PB activity space where a specific micro structure is organized for each cluster.