Baselines in Software Defect Detection

The goal of the PROMISE workshop is to create a set of re-producible software engineering experiments with the hope that researchers will repeat and improve on prior results. To show that something has improved, there must first be some defined baseline. This paper lists the baselines known for data mining results of defect detectors from static code measures and defect logs found in the PROMISE data repos-itory [6]. Many of these results have appeared previously [2] but the stratification study at the end of this paper is a new result. 1. EVALUATION CRITERIA A defect detector hunts for a signal that a software module is defect prone. In Figure 1, if a detector senses a signal, sometimes the signal is present (cell D) and sometimes it is not (cell C). Alternatively, the detector may be silent when the signal is absent (cell A) or present (cell B). If the detector registers a signal, there are two cases of interest. In one case, the detector has correctly recognized the signal. This probability of detection, or “PD”, is the ratio of detected signals, true positives, to all signals: probability detection (a.k.a. recall) = PD = D B +D In the other case, the probability of a false alarm, or “PF”, is the percentage of detections when no signal was present: probability false alarm = PF = C A+ C Another criteria of interest is the accuracy, or “Acc”, of a detector which is the number of true negatives and true positives seen over all events: accuracy = Acc