Highly significant findings in psychology: A power and effect size survey

The quality of current psychological research has been questioned because of perceived flaws in the primary methods of inquiry. As early as the late 1930s researchers began criticizing the methods by which psychological research examines hypotheses. Over the last 50 years, researchers have suggested that the American Psychological Association require additional indices to augment findings of statistical significance. Among the indices suggested are statistical power and measures of magnitude of effect size. While researchers and the APA have suggested utilizing these and other methods to supplement NHST, it seems that researchers are not currently using statistical power and effect size measures to their fullest extent. The current study examined the articles in three psychology journals to assess the current status of statistical power and effect size measures. The results of the current study suggest that about 7% of studies estimate or discuss statistical power, and about 30% calculate effect size measures. These numbers are far below the desired level of mandatory reporting of these measures. Also, when statistical power was calculated for 157 articles (45 in the Journal of Personality and Social Psychology, 57 in the Journal of Consulting and Clinical Psychology, and 55 in the Journal of Abnormal Psychology ) for 2,747 statistical tests for a total of 27,705 power calculations (power was calculated for effects beyond the normal small, medium, and large), a slight increase (above the original 1962 study and the replication in 1990) in statistical power was noted. In terms of effect size measures, a medium effect size was discovered as the average effect size across studies, which confirms previous researchers speculations about the average effect size in psychological research. It would seem that though the average effect size in the current research is of medium size, current research designs do not have sufficient statistical power to detect such an effect size. The current research should also strive to improve current statistical power survey methods to incorporate more advanced statistical methods to gain a more representative evaluation of the average effect size in psychological research