Arguments for and against the use of multiple comparison control in stochastic simulation studies

Pick up any of the standard discrete-event simulation textbooks and you will find that the output analysissection includes a note on multiple comparison control (MCC). These procedures aim to mitigate theproblem of inflating the probability of making a single type I error when comparing many simulatedscenarios simultaneously. We consider the use of MCC in stochastic simulation studies and present anargument discouraging its use in the classical sense. In particular, we focus on the impracticality ofprocedures, the benefits of common random numbers and that simulation is very different from empiricalstudies where MCC has its roots. We then consider in what instances would abandoning MCC altogetherbe problematic and what alternatives are available. We present an argument for medium to largeexploratory studies to move their attention away from classical Type I errors and instead control asubtlety different quantity: the rate of false positives amongst all ‘discoveries’