Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study

Objectives To systematically explore the methodological factors underpinning discrepancies in the pooled effect estimates from Cochrane reviews (CRs) and non-Cochrane reviews (NCRs) systematic reviews, answering the same clinical question. Study Design and Setting Quantitative and qualitative analysis of concordance in effect estimates between meta-analyses from CR and NCR matched on population, intervention, condition, and outcome. Results We identified 24 matched meta-analyses from 24 CR to 20 NCR reviews (545 randomized controlled trials [RCTs]). Compared to their CR matched-pair, pooled effects from NCR were the same in only one pair, were on average 0.12 log units (13%) higher (P = 0.012), and had a greater than twofold larger effect size in four matched-pairs. Two-thirds of CR (15/24, 70.8%) and 0/20 (0%) NCR were rated to have moderate to high confidence in their results (AMSTAR 2). Differences in pre-defined methods, including search strategy, eligibility criteria, and performance of dual screening, could explain mismatches in included studies. Disagreements in the interpretation of eligibility criteria were identified as reasons underpinning discrepant findings in 14 pairs. 23/24 meta-analyses included at least one study of its match. Only two pairs agreed on the numerical data presented for the same studies. An assessment of 50% of discrepant studies (n = 45) showed that reasons for differences in extracted data could be identified in 15 studies. Conclusion On average, meta-analyses from NCR reported higher effect estimates compared with meta-analyses from CR answering the same clinical question. Methodological and author judgments and performance are key aspects underpinning poor overlap of included studies and discrepancies in reported effect estimates. The potential impacts on health care policy and clinical practice are far-reaching but still remain unknown. Reinforcing awareness and scrutiny of application of reporting guidelines and improvements in protocol registration are needed