Controlling the rate of type I error over a large set of statistical tests

When many tests of signi cance are examined in a research investigation with procedures that limit the probability of making at least one Type I error—the so-called familywise techniques of control—the likelihood of detecting effects can be very low. That is, when familywise error controlling methods are adopted to assess statistical signi cance, the size of the critical value that must be exceeded in order to obtain statistical signi cance can be extremely large when the number of tests to be examined is also very large. In our investigation we examined three methods for increasing the sensitivity to detect effects when family size is large: the false discovery rate of error control presented by Benjamini and Hochberg (1995), a modi ed false discovery rate presented by Benjamini and Hochberg (2000) which estimates the number of true null hypotheses prior to adopting false discovery rate control, and a familywise method modi ed to control the probability of committing two or more Type I errors in the family of tests examined—not one, as is the case with the usual familywise techniques. Our results indicated that the level of signi cance for the two or more familywise method of Type I error control varied with the testing scenari