Systematic reviews of randomized controlled trials in literacy research : methodological challenges

Introduction: In this item a 'tertiary' review of systematic reviews in literacy learning is presented. It explores the methodological quality of the identified systematic reviews and identifies the primary data that are used for the in-depth methodological work in Item 3 on the two main threats to the validity of systematic reviews: publication bias and design bias. Background: Recent governments in the UK have introduced a number of initiatives aimed at improving the literacy levels of children. It is important, therefore, that policy and practice are informed by the most rigorous available evidence, particularly for questions of effectiveness in literacy learning. It is also important that this evidence is subjected to rigorous critical scrutiny. Methods: Systematic reviews undertaken in the field of literacy learning in English in the years between 1983 and 2003 were searched for, located and quality assessed. The scope of the review was limited to systematic reviews of experimental research evaluating literacy interventions with quantifiable literacy outcome measures in English as a first (not second or additional) language and focusing on children and young people in school settings up to the age of 18. Results: A total of 14 systematic reviews containing meta-analyses and meeting all the inclusion criteria were included in the tertiary review. The following data were extracted from the reviews: literacy interventions, outcomes evaluated and effect sizes. The quality of the reviews was examined using an adaptation of the QUORUM statement. Overall the quality of the meta-analyses included in this tertiary review was good. When examining the effect sizes of randomized controlled trials (RCTs) and controlled trials (CTs) separately there was no clear pattern as to whether the RCTs produced a larger or smaller effect size than the CTs. Discussion: Overall the quality of the meta-analyses included in this tertiary review was good. The QUORUM checklist seemed to perform well for the appraisal of educational meta-analyses. All the reviews clearly stated their research question, and their methods of searching for and selecting included studies. Most studies described their data extraction and used some form of quality assessment of included studies. On the other hand, some reviews did have notable methodological weaknesses. Six of the 14 studies did not make an assessment of publication bias, which is potentially a major threat to the validity of any systematic review. In addition, six studies did not provide evidence for reviewer agreement when synthesising the data. There is, therefore, some room for improvement in the methodological quality of systematic reviews in literacy learning. Conclusions: A number of reviews in this tertiary review are judged to be of sufficiently high quality to provide reliable evidence for the effectiveness of literacy interventions