How Not To Do It

We give some dos and don'ts for those analysing algorithms experimentally. We illustrate these with many examples from our own research. Where we have not followed these maxims, we have suffered as a result. 1 Introduction The empirical study of algorithms is a relatively immature field with many technical and scientific problems. We shall not address the many technical problems (e.g. determining appropriate performance measures and representative problem samples) but will instead look at some of the scientific problems. We support the calls of McGeoch [10], Hooker [9] and others for a more scientific approach to the empirical study of algorithms. Our contribution here is colloquial. With time and more such tales from the front-line, we hope that standard experimental practices in the study of algorithms will develop which represent "good science". Our own research has been into satisfiability (SAT), concentrating on two main areas. First, we have looked at hill-climbing procedures fo..