Efficient probabilistic classification vector machine with incremental basis function selection

Abstract—Probabilistic classification vector machine (PCVM) [5] is a sparse learning approach aiming to address the stability problems of relevance vector machine (RVM) for classification problems. Since PCVM is based on the Expectation Maximization (EM) algorithm, it suffers from sensitivity to initialization, convergence to local minima, and the limitation of Bayesian estimation making only point estimates. Another disadvantage is that PCVM was not efficient for large data sets. To address these problems, this paper proposes an efficient probabilistic classification vector machine (EPCVM) by sequentially adding or deleting basis functions according to the marginal likelihood maximization for efficient training. Due to the truncated prior used in EPCVM, two approximation techniques, i.e. Laplace approximation and expectation propagation, have been used to implement EPCVM to obtain full Bayesian solutions. We have verified Laplace approximation and expectation propaga-tion with a hybrid Monte Carlo approach. The generalization performance and computational effectiveness of EPCVM are extensively evaluated. Theoretical discussions using Rademacher complexity reveal the relationship between the sparsity and the generalization bound of EPCVM