Statistical Methodology for Comparison of SAT Solvers

Evaluating improvements to modern SAT solvers and comparison of two arbitrary solvers is a challenging and important task. Relative performance of two solvers is usually assessed by running them on a set of SAT instances and comparing the number of solved instances and their running time in a straightforward manner. In this paper we point to shortcomings of this approach and advocate more reliable, statistically founded method-ologies that could discriminate better between good and bad ideas. We present one such methodology and illustrate its application.