Numerical Replication of Computer Simulations: Some Pitfalls and How To Avoid Them * Theodore C. BeldingCenter for the Study of Complex Systems

A computer simulation, such as a genetic algo-rithm, that uses IEEE standard floating-point arithmetic may not produce exactly the sameresults in two different runs, even if it is rerun on the same computer with the same input andrandom number seeds. Researchers should not simply assume that the results from one runreplicate those from another but should verify this by actually comparing the data. However,researchers who are aware of this pitfall can reliably replicate simulations, in practice. Thispaper discusses the problem and suggests solutions. 1 INTRODUCTION When we perform computer simulations, such as ge-netic algorithms, it is often useful to be able to replicate a run exactly, so that those results of the second runthat we care about are exactly the same as those of the first. This kind of replication is called numerical repli-cation [1]. For instance, if we notice a strange result in a run, it is useful to be able to redo the run exactly, usingthe same parameter settings and random number seeds, but this time collecting additional data or perturbingthe course of the run in order to test hypotheses about what is causing the strange results. Numerical replica-tion can also be used to verify that experimental results are not due to a bug or human error, and to performregression testing after making changes to a program. However, a program that uses IEEE standard floating-point arithmetic [13, 6] may produce different results on two different computers, even if the same input andrandom number seeds are used. In fact, there is no guarantee that it will produce the same results whe