Lessons from comparing narrative synthesis and meta-analysis in a systematic review

Background Narrative synthesis approaches have been faulted for vote-counting (in which effectiveness is determined by tallying direction and statistical significance of study results) and lack of transparency. Yet narrative syntheses can also highlight trends and explanations for findings in a way that statistical meta-analysis cannot. Debate about the absence of meta-analysis in situations where it could have been done continues in the methodological literature. Using a systematic review of the effects of plain packaging of tobacco products, we compared narrative syntheses and new meta-analyses. We chose this review because its findings, which informed UK public health policy, have been misrepresented by the tobacco industry. Methods We revisited a published systematic review on plain packaging of tobacco products. We compared the findings from the narrative synthesis originally used in the review that accounted for study quality with the findings from a new multilevel meta-analysis that included all available effect sizes. We compared them in terms of their inclusiveness, approach to heterogeneity, and overall findings. Findings Although 21 studies (n=27 166) were included in the narrative synthesis, we included seven studies in our new meta-analysis (56 effect sizes, n=5365). The narrative synthesis found that the 21 studies were “highly consistent” in the decreased attractiveness of plain packaging compared with branded packaging, primarily on the basis of direction and significance of effects as reported in the 21 studies. The pooled effect size for the seven studies in the meta-analysis demonstrated that plain packs were less attractive (Cohen's d=–0·59, 95% CI −0·71 to −0·47). Whereas the narrative synthesis highlighted key generalities across subgroups and study designs, the meta-analysis highlighted possible effect modification by tobacco brand. Interpretation This study adds to existing debates by showing the contribution and implications of each synthesis method, especially with respect to opportunities for exploring heterogeneity. The findings have implications for the credibility of different approaches. Even when narrative synthesis is preferred, a meta-analysis and forest plot can help generate additional hypotheses, test and confirm hypotheses, and understand heterogeneity. In this particular case, use of multilevel meta-analysis helped to make clear aspects of statistical heterogeneity that narrative synthesis alone might not have detected