Psychovisual evaluation of image segmentation algorithms.

Evaluation plays an important role in the advancement of any field. In computer vision, unsupervised segmentation algorithms, although of great interest, often suffer from lack of a well-defined goal and/or explicit ground truth data, thus rendering evaluation difficult. This paper presents a novel method for evaluating such algorithms using a database for which ground truth data is not explicitly available. Unlike methods of evaluation that rely on the existence or creation of explicit ground truth data, the proposed evaluation procedure subjects human observers to a psychovisual test comparing the results of different segmentation algorithms. The test is designed to answer two main questions: does consensus about a ‘best’ segmentation exist, and if it does, what do we learn about segmentation schemes? The results confirm that human subjects are consistent in their judgements, thus allowing meaningful evaluation. The relevance of the procedure for the evaluation of CBIR systems is discussed