Quantification of improvement in risk prediction models

Thesis (Ph.D.)--Boston University, 2012.PLEASE NOTE: Boston University Libraries did not receive an Authorization To Manage form for this thesis or dissertation. It is therefore not openly accessible, though it may be available by request. If you are the author or principal advisor of this work and would like to request open access for it, please contact us at open-help@bu.edu. Thank you.The identification of new factors in modeling the probability of a binary outcome is the main challenge for statisticians and clinicians who want to improve risk prediction. This motivates researchers to search for measures that quantify the performance of new markers. There are many commonly used measures that assess the performance of the binary outcome model: logistic R-squares, discrimination slope. area under the ROC (receiver operating characteristic) curve (AUC) and Hosmer-Lemeshow goodness of fit chi-square. However, metrics that work well for model assessment may not be as good for quantifying the usefulness of new risk factors, especially when we add a new predictor to a well performing baseline model. The recently proposed new measures of improvement, the Integrated Discrimination Improvement (IDI) - difference between discrimination slopes - and the Net Reclassification Improvement (NRI), directly address the question of model performance and take it beyond the simple statistical significance of a new risk factor. Since these two measures are new and have not been studied as extensively as the older ones, a question of their interpretation naturally arises. In our research we propose meaningful interpretations to the new measures as well as extensions of these measures that address some of their potential shortcomings. Following the derivation of the maximum R-squared by Nagelkerke, we show how the IDI, which depends on the event rate, could be rescaled by its hypothetical maximum value to reduce this dependence. Furthermore, the IDI metric assumes a uniform distribution for all risk cut-offs. Application of clinically important thresholds prompted us to derive a metric that includes a prior distribution for the cut-off points and assigns different weights to sensitivity and specificity. Similarly, we propose the maximum and rescaled NRI. The latter is based on counting the number of categories by which the risk of a given person moved and guarantees that reclassification tables with equal marginal probabilities will lead to a zero NRI. All developments are investigated employing numerical simulations under the assumption of normality and varying effect sizes of the associations. We also illustrate the proposed concepts using examples from the Framingham Heart Study.2031-01-0