Just ask a human? – Controlling Quality in Relational Similarity and Analogy Processing using the Crowd

Abstract: Advancing semantically meaningful and human-centered interaction par-adigms for large information systems is one of the central challenges of current in-formation system research. Here, systems which can capture different notions of ‘similarity ’ between entities promise to be particularly interesting. While simple en-tity similarity has been addresses numerous times, relational similarity between en-tities and especially the closely related challenge of processing analogies remain hard to approach algorithmically due to the semantic ambiguity often involved in these tasks. In this paper, we will therefore employ human workers via crowd-sourc-ing to establish a performance baseline. Then, we further improve on this baseline by combining the feedback of multiple workers in a meaningful fashion. Due to the ambiguous nature of analogies and relational similarity, traditional crowd-sourcing quality control techniques are less effective and therefore we develop novel tech-niques paying respect to the intrinsic consensual nature of the task at hand. These works will further pave the way for building true hybrid systems with human work-ers and heuristic algorithms combining their individual strength.