The influence of spectrum (case-mix) on diagnostic test accuracy

This thesis assessed the degree to which the technique of meta-analysis can provide insight into spectrum effects through comparing study results between studies (or between subgroups within studies). Chapter 1 introduced the concept of diagnostic accuracy as the means by which diagnostic tests can be evaluated and also introduced the idea that diagnostic tests can operate differently according to spectrum-related characteristics. It was hypothesised that meta-analysis may provide the best available tool to identify the extent to which various the sources of heterogeneity, including spectrum, can affect test accuracy. Chapter 2 explained four methods of meta-analysis that allow for variability in threshold and for variation in DOR with threshold. Only the so-called 'advanced' models possess the characteristics of an 'optimal' meta-analytic method. Chapter 3 reported a methodological review of how heterogeneity has been examined in existing systematic reviews of diagnostic test accuracy. Less than optimal methods of meta analysis that do not allow for threshold effects have been commonly employed. Spectrum related variables were commonly investigated as potential sources of heterogeneity and 'statistically significant' results often reported. The few reviews using the advanced models of meta-analysis showed overall improved systematic review methods and were more likely to have considered spectrum-related characteristics. Chapter 4 reported a detailed case study comparing the four meta-analytic methods on a large dataset of tests for the detection of tuberculosis. The main observations arsing from these analyses were further explored in Chapter 5 using data obtained from a large sample of previously published systematic reviews of diagnostic tests and using only spectrum-related covariates. The main findings were as follows: 1. On average, weighting the Moses model by the inverse variance of the log of the DOR (SE(lnDOR)) underestimated the results of the unweighted Moses model by around 30%, with considerable disagreement between models. This underestimation is likely due to bias in the SE(lnDOR) and hence it is likely that the weighted model results are misleading. The circumstances that lead to biased SE(lnDOR) are common in diagnostic test meta-analyses therefore this form of weighting is not recommended. 2. The unweighted Moses model results were more similar to those of the HSROC model than those of the weighted Moses model, however it cannot be relied upon to approximate the results of the 'optimal' HSROC model. 3. The BVN model and the HSROC model produce almost identical results for the primary data analyses (this was investigated only in Chapter 4) 4. For the HSROC model, allowing for differences in the distribution of test results between diseased and nondiseased by covariate (shape differences) sometimes affects the conclusions that would be drawn from an analysis and sometimes not. Although the magnitude of differences between groups may vary between models, the inclusion of a shape interaction term does not necessarily change the strength of evidence for differences in accuracy. It is not clear whether potential differences in the distributions of test results (differences in shape) should be routinely modelled or whether the more simple parallel curve approach will generally suffice. The optimal approach for the investigation of heterogeneity requires further investigation 5. Finally, strong evidence of effects from spectrum-related characteristics on at least one model parameter were identified by the parallel or crossing curve HSROC model for over half of the investigations conducted in Chapter 5 (32/50). This could have considerable implications for the use of tests in practice. The advanced methods of meta-analysis show promise in enabling the detection of clinically important spectrum effects. However, one of the ongoing challenges in the investigation of heterogeneity, and especially spectrum, in systematic reviews are limitations in the primary study data.