Rejection threshold optimization using 3D ROC curves: Novel findings on biomedical datasets

Reject option is introduced in classification tasks to prevent potential misclassifications. Although optimization of error-reject trade-off has been widely investigated, it is shown that error rate itself is not an appropriate performance measure, when misclassification costs are unequal or class distributions are imbalanced. ROC analysis is proposed as an alternative approach to performance evaluation in terms of true positives (TP) and false positives (FP). Considering classification with reject option, we need to represent the tradeoff between TP, FP and rejection rates. In this paper, we propose 3D ROC analysis to determine the optimal rejection threshold as an analogy to decision threshold optimization in 2D ROC curves. We have demonstrated our proposed method with Naive Bayes classifier on Heart Disease dataset and validated the efficiency of the method on multiple datasets from UCI Machine Learning Repository. Our experiments reveal that classification with optimized rejection threshold significantly improves true positive rates in biomedical datasets. Furthermore, false positive rates remain the same with rejection rates below 10% on average. © 2021, Ismail Saritas. All rights reserved