Harvard

We examine designs for crowdsourcing contests, where par-ticipants compete for rewards given to superior solutions of a task. We theoretically analyze tradeoffs between the expecta-tion and variance of the principal’s utility (i.e. the best solu-tion’s quality), and empirically test our theoretical predictions using a controlled experiment on Amazon Mechanical Turk. Our evaluation method is also crowdsourcing based and re-lies on the peer prediction mechanism. Our theoretical anal-ysis shows an expectation-variance tradeoff of the principal’s utility in such contests through a Pareto efficient frontier. In particular, we show that the simple contest with 2 authors and the 2-pair contest have good theoretical properties. In con-trast, our empirical results show that the 2-pair contest is the superior design among all designs tested, achieving the high-est expectation and lowest variance of the principal’s utility