A statistically correct methodology to compare metaheuristics in resource-constrained Project Scheduling

The paper presents an appropriate method for comparing different heuristics or metaheuristics. The necessity of comparison is unquestionable, because the evaluation of competing techniques is an actual topic and results are highly important. In resource-constrained project scheduling the adaptation of the appropriate elements of the very rigorous protocol is necessary just as in the case of testing a new drug, or comparing the effects of different drugs. The problem is widely discussed by the heuristic community and thus it cannot be only connected to the resource-constrained project scheduling problem. Until now the result of these researches are without real detectable results. When, for example, we use stochastic searching methods in the resource-constrained makespan minimization (namely heuristics or metaheuristics with several tunable parameters and starting seeds), then the usual presentation practice: ‘one problem — one result’ is extremely far from the fair comparison. From statistical point of view, the minimal requirement of the fair comparison is a small-sample for each investigated approach and an appropriate nonparametric-small-sample-test according to the experimental design theory and the very slowly changing result presentation standard. The viability and efficiency of the proposed statistically correct and bias-free methodology is demonstrated using the hardest subset of the well-known J30 set from Project Scheduling Problem Library to decide, which of two selection operators is better in a harmony search metaheuristic frame. An appropriate test of tests, which is able to analyze from statistical point of view the combination of independent tests, or in other words, the result on the test set as a whole, is under development, and will be presented in a forthcoming paper