Three strategies for the critical use of statistical methods in psychological research

One of the earliest criticisms to the null hypothesis statistical significance testing (NHST) approach in psychology was put forward by J. Cohen (1994). Not only did Cohen criticize NHST but also the tendency of psychological researchers to utilize this method in an uncritical manner. In this article, we do not provide a critique over NHST (for a review of criticisms to NHST and a proposed solution for psychology, see Wagenmakers, 2007); rather, inspired by Cohen’s call, we propose three strategies (none of which involves NHST) to encourage a critical use of statistical methods. The strategies are (1) visual representation of cognitive processes and predictions, (2) visual representation of data distributions and choice of the appropriate distribution for analysis, and (3) model comparison. The first strategy aims to provide explicit information about the research design and the psychological theory that were considered in order to derive the main cognitive predictions. The second strategy aims to apply the most appropriate, known formal distribution to the observed data. The third strategy generates a plurality of models and selects the most suitable. The three strategies have been used in the past, so we are not claiming originality. Rather, the goal of this article is to propose that researchers use these three strategies together and give an example of how this would work. We first present the three strategies, then we provide a working example, and finally we discuss the implications of their use