Learning from multiple annotators with gaussian processes

Abstract. In many supervised learning tasks it can be costly or infea-sible to obtain objective, reliable labels. We may, however, be able to obtain a large number of subjective, possibly noisy, labels from multi-ple annotators. Typically, annotators have different levels of expertise (i.e., novice, expert) and there is considerable diagreement among an-notators. We present a Gaussian process (GP) approach to regression with multiple labels but no absolute gold standard. The GP framework provides a principled non-parametric framework that can automatically estimate the reliability of individual annotators from data without the need of prior knowledge. Experimental results show that the proposed GP multi-annotator model outperforms models that either average the training data or weigh individually learned single-annotator models.