Frequency and impact of confounding by indication and healthy vaccinee bias in observational studies assessing influenza vaccine effectiveness: a systematic review

Background: Evidence on influenza vaccine effectiveness (VE) is commonly derived from observational studies. However, these studies are prone to confounding by indication and healthy vaccinee bias. We aimed to systematically investigate these two forms of confounding/bias. Methods: Systematic review of observational studies reporting influenza VE and indicators for bias and confounding. We assessed risk of confounding by indication and healthy vaccinee bias for each study and calculated ratios of odds ratios (crude/adjusted) to quantify the effect of confounder adjustment. VE-estimates during and outside influenza seasons were compared to assess residual confounding by healthy vaccinee effects. Results: We identified 23 studies reporting on 11 outcomes. Of these, 19 (83 %) showed high risk of bias: Fourteen due to confounding by indication, two for healthy vaccinee bias, and three studies showed both forms of confounding/bias. Adjustment for confounders increased VE on average by 12 % (95 % CI: 7–17 %; all-cause mortality), 9 % (95 % CI: 4–14 %; all-cause hospitalization) and 7 % (95 % CI: 4–10 %; influenza-like illness). Despite adjustment, nine studies showed residual confounding as indicated by significant off-season VE-estimates. These were observed for five outcomes, but more frequently for all-cause mortality as compared to other outcomes (p = 0.03) and in studies which indicated healthy vaccinee bias at baseline (p = 0.01). Conclusions: Both confounding by indication and healthy vaccinee bias are likely to operate simultaneously in observational studies on influenza VE. Although adjustment can correct for confounding by indication to some extent, the resulting estimates are still prone to healthy vaccinee bias, at least as long as unspecific outcomes like all-cause mortality are used. Therefore, cohort studies using administrative data bases with unspecific outcomes should no longer be used to measure the effects of influenza vaccination