A comparison of statistical models for multiple treatment groups meta-analysis

Multivariate meta-analytic regression (MMR) is the recommended approach for conducting fixed effects multiple treatment groups (multivariate) meta-analysis; however, this method has not been widely adopted by substantive researchers. The purpose of this study was to compare the MMR approach with a structural equation modeling (SEM) approach to testing hypotheses about multiple treatment groups. The critical difference between these approaches concerns the method of ascertaining within-study covariance between effect sizes (i.e., covariance between treatment effects). I compared the two methods in a Monte Carlo simulation and a small real substantive example; focused on the correlation between treatment effects; hypothesis tests of the difference between treatment effects: and tests of homogeneity of effect sizes. First, the results of the Monte Carlo study indicated that MMR tended to underestimate the correlation between treatment effects: whereas SEM tended to overestimate the correlation between treatment effects (slightly). Second, differences in the correlation estimates resulted in different patterns of relative bias for the standard error of the contrast of treatment effects (e.g., MMR generally exhibited lower bias than SEM). Consequently, the MMR Z test generally acceptable Type-I error rates; where the SEM Z test error rates were too high. However, the SEM equality constraint approach (i.e., χ2Δ test) exhibited acceptable error rates under most conditions. Third, both MMR and SEM tests of homogeneity of effect sizes generally exhibited adequate power and Type-I error rates; whereas the SEM random effects test did not exhibit adequate rates under most conditions. Fourth, and finally, the results of the small substantive multiple treatment groups meta-analysis revealed that the Monte Carlo results did not generalize drastically different conditions (i.e., control group larger than treatment groups and treatment effects of equal and opposite magnitude). In the substantive example, MMR estimated a larger correlation between treatment effects than SEM. Consequently, MMR indicated a significant difference between treatment effects: whereas SEM did not. Therefore, assuming conditions similar to the Monte Carlo study, I recommend MMR when the number of studies relatively small, but recommend SEM when the number of studies is moderate or large and when the proportion of missingness is moderate or large.