A Randomized ANOVA Procedure for Comparing Performance Curves

Three factors enter into analyses of performance curves such as learning curves: the amount of training, the learning algorithm, and performance. Often we want to know whether the algorithm affects performance, whether the effect of training on performance depends on the algorithm, and whether these effects are localized in regions of the curves. Analysis of variance is adapted to answer these questions. The carryover effects of learning violate the assumptions of parametric analysis of variance, but they are rendered harmless by a novel, randomized version of the analysis. After a brief outline of the statistical preliminaries, we present the procedure along with some examples on real learning curves, discuss power and Type I error, and give some examples of how our method can be applied to answer more advanced questions in comparing performance curves. 1 Introduction Evaluation of machine learning algorithms typically involves learning curves that plot the amount of training versus ..