Post-hoc Power Analyses for ANOVA F-tests

When an F-test for a treatment effect has a high p-value then there is little evidence for rejection of the (null) hypothesis of no treatment differences. We may not be comfortable with this result, especially if treatment differences were expected or decisions based on the test results need to be made. While the non-significant statistical test tells us that the observed differences are small relative to the background variability (as estimated by the error term) it does not tell us what ability, known as power, the test had to find differences between treatment responses. If the power is high then accepting1 the null hypothesis is reasonable, but if it is low, a clear decision cannot be made. The power of a test depends upon the size of the treatment differences under consideration. It is important to ask what treatment differences should be under consideration. For instance, when the power of the test is determined for the observed F-value it is the observed treatment differences that are under consideration. The power at this point is only of interest if they are similar in size to a practical difference. A practical difference can be chosen on the basis that treatment differences less than this value would suggest one course of action or decision while larger differences would suggest a different course of action or decision. For instance, in a trial comparing a new fertilizer to the standard one, finding a practical difference in response migh