Local Novelty Detection in Multi-class Recognition Problems

In this paper, we propose using local learning for multi-class novelty detection, a framework that we call local nov-elty detection. Estimating the novelty of a new sample is an extremely challenging task due to the large variability of known object categories. The features used to judge on the novelty are often very specific for the object in the im-age and therefore we argue that individual novelty models for each test sample are important. Similar to human ex-perts, it seems intuitive to first look for the most related images thus filtering out unrelated data. Afterwards, the system focuses on discovering similarities and differences to those images only. Therefore, we claim that it is ben-eficial to solely consider training images most similar to a test sample when deciding about its novelty. Following the principle of local learning, for each test sample a lo-cal novelty detection model is learned and evaluated. Our local novelty score turns out to be a valuable indicator for deciding whether the sample belongs to a known category from the training set or to a new, unseen one. With our lo-cal novelty detection approach, we achieve state-of-the-art performance in multi-class novelty detection on two pop-ular visual object recognition datasets, Caltech-256 and ImageNet. We further show that our framework: (i) can be successfully applied to unknown face detection using the Labeled-Faces-in-the-Wild dataset and (ii) outperforms re-cent work on attribute-based unfamiliar class detection in fine-grained recognition of bird species on the challenging CUB-200-2011 dataset. 1