The effects of clinical and statistical heterogeneity on the predictive values of results from meta-analyses

AbstractVariance between studies in a meta-analysis will exist. This heterogeneity may be of clinical, methodological or statistical origin. The last of these is quantified by the I2-statistic. We investigated, using simulated studies, the accuracy of I2 in the assessment of heterogeneity and the effects of heterogeneity on the predictive value of meta-analyses. The relevance of quantifying I2 was determined according to the likely presence of heterogeneity between studies (low, high, or unknown) and the calculated I2 (low or high). The findings were illustrated by published meta-analyses of selective digestive decontamination and weaning protocols. As expected, I2 increases and the likelihood of drawing correct inferences from a meta-analysis decreases with increasing heterogeneity. With low levels of heterogeneity, I2 does not appear to be predictive of the accuracy of the meta-analysis result. With high levels of heterogeneity, even meta-analyses with low I2-values have low predictive values. Most commonly, the level of heterogeneity in a meta-analysis will be unknown. In these scenarios, I2 determination may help to identify estimates with low predictive values (high I2). In this situation, the results of a meta-analysis will be unreliable. With low I2-values and unknown levels of heterogeneity, predictive values of pooled estimates may range extensively, and findings should be interpreted with caution. In conclusion, quantifying statistical heterogeneity through I2-statistics is only helpful when the amount of clinical heterogeneity is unknown and I2 is high. Objective methods to quantify the levels of clinical and methodological heterogeneity are urgently needed to allow reliable determination of the accuracy of meta-analyses