Improving Credibility of Machine Learner Models in

Software Engineering Given a choice, software project managers frequently prefer traditional methods of making decisions rather than rely on empirical software engineering (empirical/machine learning-based models). One reason for this choice is the perceived lack of credibility associated with these models. To promote better empirical software engineering, a series of experiments are conducted on various NASA datasets to demonstrate the importance of assessing the ease/difficulty of a modeling situation. Each dataset is divided into 3 groups, a training set, and “nice/nasty ” neighbor test sets. Using a nearest neighbor approach, “Nice neighbors ” align closest to same class training instances. “Nasty neighbors ” align to the opposite class training instances. The “Nice, ” “Nasty ” experiments average 94 and 20 percent accuracy respectively. Another set of experiments show how a 10-fold cross-validation is not sufficient in characterizing a dataset. Finally, a set of metric equations is proposed for improving the credibility assessment of empirical/machine learning models