RESEARCH ARTICLE Randomization Does Not Help Much, Comparability Does

According to R.A. Fisher, randomization “relieves the experimenter from the anxiety of con-sidering innumerable causes by which the data may be disturbed. ” Since, in particular, it is said to control for known and unknown nuisance factors that may considerably challenge the validity of a result, it has become very popular. This contribution challenges the received view. First, looking for quantitative support, we study a number of straightforward, mathe-matically simple models. They all demonstrate that the optimism surrounding randomization is questionable: In small to medium-sized samples, random allocation of units to treatments typically yields a considerable imbalance between the groups, i.e., confounding due to ran-domization is the rule rather than the exception. In the second part of this contribution, the reasoning is extended to a number of traditional arguments in favour of randomization. This discussion is rather non-technical, and sometimes touches on the rather fundamental Fre-quentist/Bayesian debate. However, the result of this analysis turns out to be quite similar: While the contribution of randomization remains doubtful, comparability contributes much to a compelling conclusion. Summing up, classical experimentation based on sound back-ground theory and the systematic construction of exchangeable groups seems to be advisable. 1 The logic of the experiment Randomization, the allocation of subjects to experimental conditions via a random procedure, was introduced by eminent statistician R.A. Fisher [1]. Arguably, it has since become the most important statistical technique. In particular, statistical experiments are defined by the use of randomization [2, 3], and many applied fields, such as evidence based medicine, draw a basic distinction between randomized and non-randomized evidence. In order to explain randomization’s eminent role, one may refer to the logic of the experi-ment, largely based on J. S. Mill’smethod of difference[4]: If one compares two groups of sub-jects (Treatment T versus Control C, say) and observes a salient contrast in the end (e.g