A Benchmark for Change Prediction

The goal of change prediction is to help developers by recommending program entities that will have to be changed alongside the entities currently being changed. To evaluate their accuracy, current change prediction approaches use data from versioning systems such as CVS or Subversion. However, as these data sources are not very accurate, they do not provide a valid basis for an objective evaluation of change prediction approaches. We propose a benchmark for an objective evaluation of change prediction approaches based on fine-grained change data recorded from IDE usage. Moreover, the change prediction approaches themselves can use the more accurate data to fine-tune their prediction. We present an evaluation procedure and use it to evaluate several change prediction approaches, both our own and from the literature, and report on the results. Our results show that using fine-grained change data significantly improves the overall accuracy of change prediction approaches.