Ar-boost: Reducing overfitting by a robust data-driven regularization strategy

Abstract. We introduce a novel, robust data-driven regularization strat-egy called Adaptive Regularized Boosting (AR-Boost), motivated by a desire to reduce overfitting. We replace AdaBoost’s hard margin with a regularized soft margin that trades-off between a larger margin, at the expense of misclassification errors. Minimizing this regularized exponen-tial loss results in a boosting algorithm that relaxes the weak learning assumption further: it can use classifiers with error greater than 1 2. This enables a natural extension to multiclass boosting, and further reduces overfitting in both the binary and multiclass cases. We derive bounds for training and generalization errors, and relate them to AdaBoost. Fi-nally, we show empirical results on benchmark data that establish the robustness of our approach and improved performance overall.