Application of Treatment Thresholds to Diagnostic-test Evaluation: An Alternative to the Comparison of Areas under Receiver Operating Characteristic Curves

Diagnostic tests are often evaluated by comparison of the areas under receiver op-erating charactenstic (ROC) curves. In this study the authors compared this approach with a more direct method that takes into account consequences of a diagnosis. Data from a prospective study of diagnosis of pulmonary embolism were used for a moti-vating example. Using multivariable logistic regression analysis, three diagnostic mod-els were built and compared based on their ROC curves. Although model 1 (0.706) and model 2 (0.702) had the same ROC-curve area, they performed differently when risks and benefits of subsequent decisions were considered by applying the treatment probability threshold. Models 1 and 3 (0.611) had substantially different ROC-curve areas but performed similarly taking into account the therapeutic consequences. This demonstrates that comparison of diagnostic tests using the areas under the ROC curves may lead to erroneous conclusions about therapeutic usefulness. To corre-spond to daily practice, it would be more appropriate to also consider the clinical im-plications in evaluating diagnostic tests. This is made feasible by explicit definition and application of a treatment threshold. Key words: benefits and risks; clinical relevance