Trick or Treat: Putting Peer Prediction to the Test

Collecting subjective information from multiple parties is a common problem in collective intelligence. However, incen-tivizing truthful reports is difficult when there is no ground truth to verify the reports against. Peer prediction mecha-nisms use collected reports alone to provide good theoreti-cal incentives for truthful reporting, but make assumptions that are difficult to satisfy in the real world. They also ad-mit uninformative equilibria where coordinating participants provide no useful information. Using a multiplayer, real-time repeated game, we conduct the first controlled online experi-ment of a peer prediction method. Our results show that play-ers learn to adopt more profitable strategies through repeated use of the mechanism, and that there is a distinct incentive for participants to converge to the uninformative equilibria.