Detecting Outliers and Influential Data Points in Receiver Operating Characteristic (ROC) Analysis

Receiver operating characteristic (ROC) studies and analyses are often used to evaluate medical tests and are very useful in the field of radiology to evaluate a single diagnostic imaging system, to compare the accuracy of two or more diagnostic imaging systems, or to assess observer performance. There have been many refinements in the development of different ROC type study designs and the corresponding statistical analysis. These methods have become increasingly important and ROC methods are the principal approach for evaluating imaging technologies and/or observer performances. The systems that are often evaluated using ROC methodology include digital and radiographic images of the chest and breast. An improved method of evaluating diagnostic imaging systems contributes to the development of better diagnostic methods; hence, improving imaging systems for diagnoses of breast and lung cancer would have major public health significance. In our work with observer performance studies, in which receiver operating characteristic (ROC) analysis is used, we have noted that some contributions of readers and cases can substantially alter the conclusions of the analysis. To the best of our knowledge, to date there is no statistical test cited in the statistical literature that addresses the detection and influence of outliers on the estimate of the area under the ROC curve. Evaluating outliers may be especially important for the ROC model since subtle (difficult) cases have the potential for being missed by a reader (e.g. a difficult positive case is rated as an unquestionably negative case), and can have a considerable influence on the estimated area under the ROC curve, especially if the study has a small set of cases. Therefore, we believe it is important to develop a method for detecting and measuring the influence of outliers for ROC models. The development of this method will involve deriving a test statistic for outliers based on the jackknife influence values and conducting a preliminary validation of the test.