How to improve the statistical power of the 10-fold cross validation scheme in recommender systems

At this stage development of recommender systems (RS), an evaluation of competing approaches (methods) yielding similar performances in terms of experiment reproduction is of crucial importance in order to direct the further devel-opment toward the most promising direction. These com-parisons are usually based on the 10-fold cross validation scheme. Since the compared performances are often simi-lar to each other, the application of statistical significance testing is inevitable in order to not to get misled by ran-domly caused differences of achieved performances. For the same reason, to reproduce experiments on a different set of experimental data, the most powerful significance testing should be applied. In this work we provide guidelines on how to achieve the highest power in the comparison of RS and we demonstrate them on a comparison of RS performances when different variables are contextualized