Dwelling on the Negative: Incentivizing Effort in Peer Prediction

Agents are asked to rank two objects in a setting where effort is costly and agents differ in quality (which is the probabil-ity that they can identify the correct, ground truth, ranking). We study simple output-agreement mechanisms that pay an agent in the case she agrees with the report of another, and po-tentially penalizes for disagreement through a negative pay-ment. Assuming access to a quality oracle, able to determine whether an agent’s quality is above a given threshold, we de-sign a payment scheme that aligns incentives so that agents whose quality is above this threshold participate and invest effort. Precluding negative payments leads the expected cost of this quality-oracle mechanism to increase by a factor of 2 to 5 relative to allowing both positive and negative pay-ments. Dropping the assumption about access to a quality oracle, we further show that negative payments can be used to make agents with quality lower than the quality threshold choose to not to participate, while those above continue to participate and invest effort. Through the appropriate choice of payments, any design threshold can be achieved. This self-selection mechanism has the same expected cost as the cost-minimal quality-oracle mechanism, and thus when using the self-selection mechanism, perfect screening comes for free