Making Inferences with Small Numbers of Training Sets

This paper discusses a potential methodological problem with empirical studies assessing project effort prediction systems. Frequently a hold-out strategy is deployed so that the data set is split into a training and a validation set. Inferences are then made concerning the relative accuracy of the different prediction techniques under examination. Typically this is done on very small numbers of sampled training sets