An AUC-based permutation variable importance measure for random forests

The random forest (RF) method is a commonly used tool for classi-fication with high dimensional data as well as for ranking candidate predictors based on the so-called random forest variable importance measures (VIMs). However the classification performance of RF is known to be suboptimal in case of strongly unbalanced data, i.e. data where response class sizes differ considerably. Suggestions were made to obtain better classification performance based either on sampling procedures or on cost sensitivity analyses. However to our knowl-edge the performance of the VIMs has not yet been examined in the case of unbalanced response classes. In this paper we explore the per-formance of the permutation VIM for unbalanced data settings and introduce an alternative permutation VIM based on the area under the curve (AUC) that is expected to be more robust towards class imbalance. We investigated the performance of the standard permuta-tion VIM and of our novel AUC-based permutation VIM for different class imbalance levels using simulated data and real data. The re-sults suggest that the standard permutation VIM loses its ability to discriminate between associated predictors and predictors not associ-ated with the response for increasing class imbalance. It is outper-formed by our new AUC-based permutation VIM for unbalanced data settings, while the performance of both VIMs is very similar in the case of balanced classes. The new AUC-based VIM is implemented in the R package party for the unbiased RF variant based on condi-tional inference trees. The codes implementing our study are available from the companion website