Learning with marginalized corrupted features

The goal of machine learning is to develop predictors that generalize well to test data. Ideally, this is achieved by training on very large (infinite) training data sets that cap-ture all variations in the data distribution. In the case of finite training data, an effec-tive solution is to extend the training set with artificially created examples—which, how-ever, is also computationally costly. We pro-pose to corrupt training examples with noise from known distributions within the expo-nential family and present a novel learning algorithm, called marginalized corrupted fea-tures (MCF), that trains robust predictors by minimizing the expected value of the loss function under the corrupting distribution— essentially learning with infinitely many (cor-rupted) training examples. We show empiri-cally on a variety of data sets that MCF clas-sifiers can be trained efficiently, may general-ize substantially better to test data, and are more robust to feature deletion at test time. 1