Modeling annotator expertise: Learning when everyone knows a bit of something

Supervised learning from multiple labeling sources is an increasingly important problem in machine learning and data mining. This paper develops a probabilistic approach to this problem when annotators may be unre-liable (labels are noisy), but also their exper-tise varies depending on the data they ob-serve (annotators may have knowledge about different parts of the input space). That is, an annotator may not be consistently accu-rate (or inaccurate) across the task domain. The presented approach produces classifica-tion and annotator models that allow us to provide estimates of the true labels and an-notator variable expertise. We provide an analysis of the proposed model under vari-ous scenarios and show experimentally that annotator expertise can indeed vary in real tasks and that the presented approach pro-vides clear advantages over previously intro-duced multi-annotator methods, which only consider general annotator characteristics.