DEVELOPMENT AND COMPARISON OF DIFFERENT METHODS OF EVALUATING FREE-RESPONSE ROC SYSTEMS

Receiver Operating Characteristic (ROC) analysis has been widely used to evaluate diagnostic systems since the 1970s. In diagnostic imaging the decision task often needs the radiologist to locate the specific region on a subject that actually contains the abnormality. A Free-Response ROC experiment has been more and more accepted for evaluating this type of a diagnostic task. It entails detecting and marking the locations of all suspected abnormalities, as well as indicating a level of suspicion regarding the specific abnormality at each marked location. Several existing approaches of analyzing FROC data used the maximum rating to represent the multiple responses of a subject and then applied an analysis in an ROC concept to summarize the diagnostic system's discriminative ability in a randomly selected pair of actually negative and actually positive subjects. This dissertation proposes and evaluates new methods of subject-based discriminative ability by considering approaches based on the average of multiple ratings and approaches based on the stochastic order. Indices are also formulated by improving the JAFROC-type indices in literature, in order to summarize the diagnostic performance with correct location information. We also propose new indices that can penalize and reward for the number of correct and incorrect marks on the subjects. Asymptotic procedures are developed to compare the discriminative ability between two FROC systems. These asymptotic approaches are then extended to the multi-reader setting, taking into consideration the correlation and heterogeneity between readers. We also apply three different approaches to fit a smooth FROC curve, namely Box-Cox transformation approach, kernel smoothing approach and kernel regression approach. The public health significance of the work lies in our efforts to improve the statistical tools for evaluating medical diagnostic devices, which can help in the development of more specific and affordable diagnostic methods. Our contribution to early diagnosis could improve the timely recognition of reportable diseases