Bolstered Error Estimation

We propose a general method for error estimation that displays low variance and gen-erally low bias as well. This method is based on “bolstering ” the original empirical dis-tribution of the data. It has a direct geometric interpretation and can be easily applied to any classification rule and any number of classes. This method can be used to im-prove the performance of any error-counting estimation method, such as resubstitution and all cross-validation estimators, particularly in small-sample settings. We point out some similarities shared by our method with a previously proposed technique, known as smoothed error estimation. In some important cases, such as a linear classification rule with a Gaussian bolstering kernel, the integrals in the bolstered error estimate can be computed exactly. In the general case, the bolstered error estimate may be computed by Monte-Carlo sampling; however, our experiments show that a very small number of Monte-Carlo samples is needed. This results in a fast error estimator, which is in con-trast to other resampling techniques, such as the bootstrap. We provide an extensive simulation study comparing the proposed method with resubstitution, cross-validation