Robotics: Environmental Awareness Through Cognitive Sensor Fusion

The self-assembling micro robots developed in the European project "Replicator" are build for such a task. By using several (low-quality) modalities, these robots must be able to detect and recognize interesting objects in the environment. This thesis presents a biologically inspired cognitive sensor fusion architecture to create environmental awareness in these micro robots. This architecture consists of a bi-modal attention and multi-modal sensor fusion module. For bi-modal attention, an auditory and visual saliency detection method was developed together with a biologically inspired sensor integration process to obtain a visual-acoustic attention module. In order to fuse uni-modal information into a multi-modal percept, hierarchical structured self-organizing associative memory, as can be found in the human medial temporal lobe, is used as basis for the multi-modal sensor fusion module. This module was found to be suitable for unsupervised ongoing learning of sound and visual objects through reservoir state and SIFT keypoint clustering, respectively. Experiments conducted in a 3D simulator showed that the simulated micro robot was able to successfully perform a variety of search tasks with the cognitive sensor fusion architecture.