Class Probabilities and the ROC Curve in Changing Environments

A probabilistic classifier assigns probability scores to data examples. The ROC curve depicts the ranking performance of the clas-sifier by imposing threshold values on these probabilities and by plotting the true positive rates against the false positive rates. Once classification decisions are made, the ROC eliminates these probabilities from the per-formance analysis. In this paper, we argue that discarding these probabilities can result in a loss of information. We show that the lost information is related to changes in the domain. We propose an evaluation method, the sensibility analysis, to remedy this situa-tion. Using synthetic data, we illustrate that visualizing the sensibility values for all clas-sification thresholds depicts changes in the underlying distribution, otherwise known as concept drift. 1