1Estimation of Inferential Uncertainty in Assessing Expert Segmentation Performance from STAPLE

Abstract—The evaluation of the quality of segmentations of an image, and the assessment of intra- and inter-expert variability in segmentation performance, has long been recognized as a difficult task. For a segmentation validation task, it may be effective to compare the results of an automatic segmentation algorithm to multiple expert segmentations. Recently an Expec-tation Maximization (EM) algorithm for Simultaneous Truth and Performance Level Estimation (STAPLE) was developed to this end to compute both an estimate of the reference standard segmentation and performance parameters from a set of segmentations of an image. The performance is characterized by the rate of detection of each segmentation label by each expert in comparison to the estimated reference standard. This previous work provides estimates of performance pa-rameters, but does not provide any information regarding th