Direct Optimization for Classification with Boosting

Boosting, as one of the state-of-the-art classification approaches, is widely used in the industry for a broad range of problems. The existing boosting methods often formulate classification tasks as a convex optimization problem by using surrogates of performance measures. While the convex surrogates are computationally efficient to globally optimize, they are sensitive to outliers and inconsistent under some conditions. On the other hand, boosting\u27s success can be ascribed to maximizing the margins, but few boosting approaches are designed to directly maximize the margin. In this research, we design novel boosting algorithms that directly optimize non-convex performance measures, including the empirical classification error and margin functions, without resorting to any surrogates or approximations. We first applied this approach on binary classification, and then extended this idea to more complicated classification problems, including multi-class classification, semi-supervised classification, and multi-label classification. These extensions are non-trivial, where we have to mathematically re-formulate the optimization problem: defining new objectives and designing new algorithms that depend on the specific learning tasks. Moreover, we showed good theoretical properties of the optimization objectives, which explains why we define these objectives and how we design algorithms to efficiently optimize them. Finally, we showed experimentally that the proposed approaches display competitive or better results than state-of-the-art convex relaxation boosting methods, and they perform especially well on noisy cases