Visual recognition in art using machine learning

This thesis is concerned with the problem of visual recognition in art – such as finding the objects (e.g. cars, cows and cathedrals) present in a painting, or identifying the subject of an oil portrait. Solving this problem is extremely beneficial to art historians, who are often interested in determining when an object first appeared in a painting or how the portrayal of an object has evolved over time. It allows them to avoid the unenviable task of finding paintings for study manually. However, visual recognition of art is a challenging problem, in part due to the lack of annotation in art. A solution is to train recognition models on natural, photographic images. These models have to overcome a domain shift when applied to art. Firstly, a thorough evaluation of the domain shift problem is conducted for the task of image classification in paintings; the performance of natural image-trained and painting- trained classifiers on a fixed set of paintings are compared for both shallow (Fisher Vec- tors) and deep image representations (Convolutional Neural Networks – CNNs) to exam- ine the performance gap across domains. Then, we show that this performance gap can be ameliorated by classifying regions using detectors. We next consider the problem of annotating gods and animals on classical Greek vases, starting from a large dataset of images of vases with associated brief text descriptions. To solve this, we develop a weakly supervised learning approach to solve the correspondence problem between the descriptions and unknown image regions. Then, we study the problem of matching photos of a person to paintings of that person, in order to retrieve similar paintings given a query photo. We show that performance at this task can be improved substantially by learning with a combination of photos and paintings – either by learning a linear projection matrix common across facial identities, or by fine-tuning a CNN. Finally, we present several applications of this research. These include a system that learns object classifiers on-the-fly from images crawled off the web, and uses these to find a variety of objects in very large datasets of art. We show that this research has resulted in the discovery of over 250,000 new object annotations across 93,000 paintings on the public Art UK website.