Reconsidering Ranking-and-Selection Guarantees

This dissertation deals with the various statistical guarantees delivered by ranking-and-selection (R&S) procedures: a class of methods designed for the problem of selecting the best from among a finite number of simulated systems. Examples of such guarantees include ensuring that an optimal or near-optimal system is selected with high probability or that the expected performance gap between the selected system and the optimal system is below a specified threshold. We explore three fundamental issues concerning R&S guarantees that are of practical and theoretical interest to the simulation community. First, we discuss the shortcomings of the popular indifference-zone-inspired guarantee on the probability of correct selection (PCS) and argue that delivering a guarantee on the probability of good selection (PGS) is a more justifiable goal. We present an overview of the PGS guarantee and examine numerous techniques for proving the PGS guarantee, including sufficient conditions under which R&S procedures that deliver the IZ-inspired PCS guarantee also deliver the PGS guarantee. Second, we study Bayesian R&S guarantees, contrasting them with their frequentist counterparts and investigating the practical implications of this distinction. R&S procedures deliver Bayesian guarantees by terminating when a posterior quantity of interest - e.g., the posterior PCS or PGS - crosses some threshold. We develop several methods for improving the computational efficiency of checking this stopping rule when there are a large number of systems and demonstrate their effectiveness compared to existing approaches. Third, we study R&S guarantees for the setting in which a R&S procedure is run after a simulation-optimization search. We show that for searches that use the observed performance of explored systems to identify new systems, the simulation replications are conditionally dependent given the sequence of returned systems. We demonstrate that reusing replications taken during a search as input to a R&S procedure can result in an empirical PCS or PGS below the guaranteed threshold. Based on these negative findings, we call into question the guarantees of established R&S procedures that reuse search data