Multilevel meta-analysis of single-case experimental designs

In a single-case experimental design (SCED), a single case is studied thoroughly under at least two manipulated conditions. A SCED typically starts with a baseline phase, during which the dependent variable is measured repeatedly before any treatment is given. This phase is followed by a treatment phase, during which the dependent variable is again measured repeatedly, after a treatment or intervention started. The effect of the treatment is evaluated by comparing the scores under both conditions, in the same way as in a group comparison experimental design (GCED) study the experimental group data are compared with those of a control group. Replicating this design across cases (within or across studies) offers opportunities for generalization, but there is a clear need for analysis techniques for combining the data of multiple cases. The aim of this dissertation is to evaluate the use of multilevel meta-analysis for SCEDs. In a multilevel meta-analysis, the longitudinal aspect of the data can easily be modelled by including trends in the model, or by including autocorrelation. In addition, predictors can be included in the model to explain differences between studies or cases. In a first step of the meta-analysis, the data of each single-case time series are summarized in effect size indices, for instance an index for the immediate treatment effect and an index for the treatment effect on the trend. Next, these effect sizes are combined in a multilevel meta-analysis, with the effect sizes summarizing the observations within each case, and with cases in turn clustered in studies. This analysis results in estimates and significance tests for the overall effect, for the between-study and between-case variance and for possible moderator effects. In a first study, we empirically evaluated a three-level meta-analytic model to analyze unstandardized and standardized effect size metrics using a Monte Carlo simulation study. The model results in unbiased effect estimates, except when there are few measurement occasions per case and when standardized effect sizes are used. Therefore, in a second study, we evaluated several methods to correct for this bias when standardized effect sizes are used. Especially Hedges’ (1981) correction for small sample bias is feasible to correct for this bias. In a third study, we evaluated a method to make the SCED effect sizes comparable to GCED effect sizes, using estimates of the between- and within-case variance estimates. At least 6 homogeneous cases are needed for unbiased effect estimates. In a fourth study, real SCED studies were systematically retrieved, the characteristics of these studies were described, and raw data were retrieved and analyzed. In a fifth study, several approaches to handle multiple dependent outcomes within a study were evaluated, namely averaging effect sizes, robust variance estimation and multilevel meta-analysis. Both robust variance estimation and multilevel meta-analysis resulted in unbiased fixed effects and corresponding standard error estimates. Based on these studies, we can conclude that the multilevel meta-analysis of SCED yields promising results in the simulated situations. However, these studies are only an additional step in a fast developing field. More research is needed on modelling more complex (and more realistic) datasets, and the performance of these complex models given the small number of data that is often encountered in SCED research.status: publishe