Neural network based 2D/3D fusion for robotic object recognition

Abstract. We present a neural network based fusion approach for real-time robotic object recognition which integrates 2D and 3D descriptors in a flexible way. The presented recognition architecture is coupled to a real-time segmentation step based on 3D data, since a focus of our inves-tigations is real-world operation on a mobile robot. As recognition must operate on imperfect segmentation results, we conduct tests of recogni-tion performance using complex everyday objects in order to quantify the overall gain of performing 2D/3D fusion, and to discover where it is par-ticularly useful. We find that the fusion approach is most powerful when generalization is required, for example to significant viewpoint changes and a large number of object categories, and that a perfect segmentation is apparently not a necessary prerequisite for successful discrimination.