Evaluation Methods of Accuracy and Reproducibility for Image Segmentation Algorithms

Segmentation algorithms perform different on differernt datasets. Sometimes we want to learn which segmentation algoirithm is the best for a specific task, therefore we need to rank the performance of segmentation algorithms and determine which one is most suitable to that task. The performance of segmentation algorithms can be characterized from many aspects, such as accuracy and reproducibility. In many situations, the mean of the accuracies of individual segmentations is regarded as the accuracy of the segmentation algorithm which generated these segmentations. Sometimes a new algorithm is proposed and argued to be best based on mean accuracy of segmentations only, but the distribution of accuracies of segmentations generated by the new segmentation algorithm may not be really better than that of other exist segmentation algorithms. There are some cases where two groups of segmentations have the same mean of accuracies but have different distributions. This indicates that even if the mean accuracies of two group of segmentations are the same, the corresponding segmentations may have different accuracy performances. In addition, the reproducibility of segmentation algorithms are measured by many different metrics. But few works compared the properties of reproducibility measures basing on real segmentation data. In this thesis, we illustrate how to evaluate and compare the accuracy performances of segmentation algorithms using a distribution-based method, as well as how to use the proposed extensive method to rank multiple segmentation algorithms according to their accuracy performances. Different from the standard method, our extensive method combines the distribution information with the mean accuracy to evaluate, compare, and rank the accuracy performance of segmentation algorithms, instead of using mean accuracy alone. In addition, we used two sets of real segmentation data to demonstrate that generalized Tanimoto coefficient is a superior reproducibility measure which is insensitive to segmentation group size (number of raters), while other popular measures of reproducibility exhibit sensitivity to group size