Bootstrapping with Noise: An Effective Regularization Technique

Bootstrap samples with noise are shown to be an effective smoothness and capacity control technique for training feed-forward networks and for other statistical methods such as generalized additive models. It is shown that noisy bootstrap performs best in conjunction with weight decay regularization and ensemble averaging. The two-spiral problem, a highly non-linear noise-free data, is used to demonstrate these findings. The combination of noisy bootstrap and ensemble averaging is also shown useful for generalized additive modeling, and is also demonstrated on the well known Cleveland Heart Data [7]. Keywords: Noise Injection, Combining Estimators, Pattern Classification, Two Spiral Problem Clinical Data Analysis. 1 Introduction The bootstrap technique has become one of the major tools for producing empirical confidence intervals of estimated parameters or predictors [8]. One way to view bootstrap is as a method to simulate noise inherent in the data, and thus, increase effectively t..