Gradient Boosted Feature Selection

A feature selection algorithm should ideally satisfy four con-ditions: reliably extract relevant features; be able to iden-tify non-linear feature interactions; scale linearly with the number of features and dimensions; allow the incorpora-tion of known sparsity structure. In this work we propose a novel feature selection algorithm, Gradient Boosted Feature Selection (GBFS), which satisfies all four of these require-ments. The algorithm is flexible, scalable, and surprisingly straight-forward to implement as it is based on a modifi-cation of Gradient Boosted Trees. We evaluate GBFS on several real world data sets and show that it matches or out-performs other state of the art feature selection algorithms. Yet it scales to larger data set sizes and naturally allows for domain-specific side information