Comparison of dataset bias in object recognition benchmarks

Current research in the area of automatic visual object recognition heavily relies on testing the performance of new algorithms by using benchmark datasets. Such datasets can be based on standardized datasets collected systematically in a controlled environment (e.g., COIL-20), as well benchmarks compiled by collecting images from various sources, normally via the World Wide Web (e.g., Caltech 101). Here we test bias in benchmark datasets by separating a small area from each image such that the area is seemingly blank, and too small to allow manual recognition of the object. The method can be used to detect the existence of dataset bias in a single object recognition dataset, and compare the bias to other datasets. The results show that all tested datasets allowed classification accuracy higher than mere chance by using the small images, although the sub-images did not contain any visually interpretable information. That shows that the consistency of the images within the different classes of object recognition datasets can allow classifying the images even by algorithms that do not recognize objects. Among the tested datasets PASCAL is the dataset with the lowest observed bias, while datasets acquired in a controlled environment such as COIL-20, COIL-100, and NEC Animals are more vulnerable to bias, and can be classified by the sub-images with accuracy far higher than mere chance. Index Terms—Object recognition, performance evaluation, benchmarks, validation, computer vision, pattern recognition. I