Language-based querying of image collections on the basis of an extensible ontology

Abstract The design of a specialised query language for content based image retrieval provides a means of addressing many of the problems associated with commonly used query paradigms such as query-by-example and query-by-sketch. By basing such a language on an extensible ontology, which encompasses both high-level and low-level image properties and relations, one can go a long way towards bridging the semantic gap between user models of saliency and relevance and those employed by a retrieval system. This paper discusses these issues and illustrates the design and use of an ontological retrieval language through the example of the OQUEL query language. The retrieval process takes place entirely within the ontological domain defined by the syntax and semantics of the user query. Since the system does not rely on the pre-annotation of images with sentences in the language, the format of text queries is highly flexible. The language is also extensible to allow for the definition of higher-level terms such as 'cars', 'people', etc. on the basis of existing language constructs through the use of Bayesian inference networks. The matching process utilises automatically extracted image segmentation and classification information and can incorporate any other feature extraction mechanisms or contextual knowledge available at processing time to satisfy a given user request.