Robust Forward Algorithms via PAC-Bayes and Laplace Distributions

Laplace random variables are commonly used to model extreme noise in many fields, while systems trained to deal with such noises are often characterized by robustness properties. We introduce new learning algorithms that minimize objectives derived directly from PAC-Bayes bounds, incorporating Laplace distributions. The resulting algorithms are regulated by the Huber loss function and are robust to noise, as the Laplace distri-bution integrated large deviation of param-eters. We analyze the convexity properties of the objective, and propose a few bounds which are fully convex, two of which jointly convex in the mean and standard-deviation under certain conditions. We derive new for-ward algorithms analogous to recent boost-ing algorithms, providing novel relations be-tween boosting and PAC-Bayes analysis. Ex-periments show that our algorithms outper-form AdaBoost, L1-LogBoost [10], and Ro-bustBoost [11] in a wide range of input noise.