iCrowd: An Adaptive Crowdsourcing Framework

Crowdsourcing is widely accepted as a means for resolving tasks that machines are not good at. Unfortunately, Crowd-sourcing may yield relatively low-quality results if there is no proper quality control. Although previous studies attempt to eliminate “bad ” workers by using qualification tests, the accuracies estimated from qualifications may not be accu-rate, because workers have diverse accuracies across tasks. Thus, the quality of the results could be further improved by selectively assigning tasks to the workers who are well ac-quainted with the tasks. To this end, we propose an adaptive crowdsourcing framework, called iCrowd. iCrowd on-the-fly estimates accuracies of a worker by evaluating her perfor-mance on the completed tasks, and predicts which tasks the worker is well acquainted with. When a worker requests for a task, iCrowd assigns her a task, to which the worker has the highest estimated accuracy among all online workers. Once a worker submits an answer to a task, iCrowd analyzes her answer and adjusts estimation of her accuracies to im-prove subsequent task assignments. This paper studies the challenges that arise in iCrowd. The first is how to estimate diverse accuracies of a worker based on her completed tasks. The second is instant task assignment. We deploy iCrowd on Amazon Mechanical Turk, and conduct extensive exper-iments on real datasets. Experimental results show that iCrowd achieves higher quality than existing approaches