A Probability-Theory Based Test of the Reliability of Election Polls

Abstract: We analyze a data set of 344 polls from presidential elections in 45 states. Using a methodology first developed in an earlier paper (Bloom, 2003) we examine the reliability of the polls with the yardsticks provided by probability theory on which the concept of sampling error is based. Unlike the off-year election Senate polls examined in that paper, we find that the presidential polls are every bit as reliable as advertised. While observed error from election results in the 2002 Senate sample substantially exceeded the polls ’ reported margins of error, we find that the 2004 statewide presidential polls varied from the election results precisely as predicted by probability theory. On the strictly subjective measure of predictive accuracy, these surveys also performed better than the 2002 Senate polls. Another interesting finding, consistent with the previous paper, is that despite criticism over the use of automated interviewers, Survey USA and Rasmussen Reports provided accurate and reliable data, consistent with other polling organizations. NOTE: After presenting this paper, the authors realized they had should have accounted for rounding error in their calculations, which would have had the net effect of increasing the expected error for most polls. This was corrected in the subsequent work, “Reliable Compare