In the era of deep convolutional features: are attributes still useful privileged data?

Our answer is, if used for challenging computer vision tasks, attributes are useful privileged data. We introduce a learning framework called learning using privileged information (LUPI) to the computer vision field to solve the object recognition task in images. We want computers to be able to learn more efficiently at the expense of providing extra information during training time. In this chapter, we focus on semantic attributes as a source of additional information about image data. This information is privileged to image data as it is not available at test time. Recently, image features from deep convolutional neural networks (CNNs) have become primary candidates for many visual recognition tasks. We will therefore analyze the usefulness of attributes as privileged information in the context of deep CNN features as image representation. We explore two maximum-margin LUPI techniques and provide a kernelized version of them to handle nonlinear binary classification problems. We interpret LUPI methods as learning to identify easy and hard objects in the privileged space and transferring this knowledge to train a better classifier in the original data space. We provide a thorough analysis and comparison of information transfer from privileged to the original data spaces for two maximum-margin LUPI methods and a recently proposed probabilistic LUPI method based on Gaussian processes. Our experiments show that in a typical recognition task such as deciding whether an object is “present” or “not present” in an image, attributes do not lead to improvement in the prediction performance when used as privileged information. In an ambiguous vision task such as determining how “easy” or “difficult” it is to spot an object in an image, we show that attribute representation is useful privileged information for deep CNN image features